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Executive Summary 


The National Aeronautics and Space Administration's (NASA) Science, Engineering, Mathematics, 
and Aerospace Academy (SEMAA) project is a science enrichment program aimed at inspiring, 
engaging, and educating the nation's K-12 students in science, technology, engineering, and 
mathematics (STEM). SEMAA is designed to attract and retain students, particularly those who are 
historically underserved and underrepresented in STEM fields, through a progression of educational 
opportunities that capitalize on the exciting nature of NASA missions and its expertise, technology, 
and resources. 

SEMAA targets students in kindergarten to 12 th grade and their parents or caregivers, and offers 
three related project components: 

• A NASA-designed, hands-on, inquiry-based curriculum aligned to national science, math, and 
technology standards at each grade level and connected to research from the NASA mission 
directorates; 

• An Aerospace Education Laboratory (AEL), featuring cutting-edge aerospace technology that 
gives students experience with aeronautical and reduced-gravity simulations; and 

• A Family Cafe to promote sustained parental involvement in their child's STEM education. 

Established at a single site in 1993, SEMAA has expanded to other sites throughout the nation. Over 
the years there have been a total of 25 SEMAA sites, and at its peak in 2005, SEMAA was being 
implemented in 24 sites across the country. This evaluation focuses on the 14 currently active sites, 
although former sites are included in analysis and discussions of SEMAA implementation models and 
sustainability. 

In 2008, the National Aeronautics and Space Administration (NASA) contracted with Abt Associates 
Inc. and its subcontractors The Urban Institute and the Education Development Center, Inc. to 
design and conduct a national evaluation of SEMAA. This report presents findings from that 
evaluation, which contained both implementation and impact modules. The implementation study 
investigated how sites implement SEMAA and the contextual factors important in this 
implementation. The implementation study used data from extant sources and interviews with the 
14 currently open sites and with 6 of the 11 closed sites. The impact study investigated the short- 
term outcomes of SEMAA for students and parents. The impact study was designed as a randomized 
control trial of students in 4 th through 8 th grades (and their parents or caregivers), and involved six 
SEMAA sites. Surveys were administered to students and parents in both the treatment group and 
control group at baseline (at the time that individuals applied for the Fall 2009 SEMAA session) and 
follow-up (after the Fall 2009 SEMAA had been completed). At baseline, the study found high levels 
of interest and engagement with STEM among students and parents who applied to the Saturday 
SEMAA program. 
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The key findings of the study include: 


• The study did not reveal overall impacts of the SEMAA program on short-term outcomes, 
but among prior participants, SEMAA led to increased participation in informal science 
activities. This impact of SEMAA on prior participants aligns with SEMAA's longitudinal 
approach to engaging students in STEM. 

• Student attendance had a positive relationship with a greater desire to engage in science- 
related activities, interest in high school STEM courses, interest in college STEM coursework 
among students, and parental encouragement for students to pursue a STEM field in 
college, as well as with students' perception that their interest in informal science activities 
had increased and with lessened anxiety toward science. Parent attendance in the Family 
Cafe was also positively related to parent and child outcomes. 

• Sites engage students in SEMAA across multiple years. The recurring exposure is consistent 
with SEMAA's program theory that students should be engaged repeatedly to maintain and 
further develop an interest in and engagement with science. 

• Three models of implementation coexist— two conducted out of school hours (Saturday 
model and after-school model) and one during school hours (in-school model). The Saturday 
model is the most prevalent, but the in-school model is used to respond to increased 
demand and fundraising needs. 

• Sustainability beyond NASA support requires funding, but site fundraising has been 
insufficient to sustain the program without NASA funding, in part because of limited 
experience in this area and competing demands. 

• Monitoring data are limited and would need to be augmented in order to investigate the 
long-term outcomes of SEMAA. 

The following recommendations are made based on the findings of this study: 

• SEMAA sites should continue their existing efforts to engage students in STEM activities 
from week to week and year to year, particularly in the out of school models where 
attendance is not mandatory. 

• The SEMAA project should continue to encourage early and ongoing strategic planning at 
the sites to establish viable sustainability plans given the particulars of the sites. 

• The SEMAA project should monitor the various SEMAA models to assess their prospects for 
sustainability and to ensure that the intended science content is maintained. 

• The SEMAA project should continue to conduct grantee training and provide support related 
to fundraising and partnerships. 

• SEMAA should conduct student monitoring data collection that would facilitate the 
investigation of the long-term outcomes of SEMAA. 
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Chapter 1: Introduction 


The National Aeronautics and Space Administration's (NASA) commitment to the nation's science, 
technology, engineering, and mathematics (STEM) education is demonstrated across its education 
portfolio, which aims to strengthen the nation's future workforce, attract and retain students in 
STEM disciplines, and engage Americans in STEM and in NASA's mission (NASA, 2007a). As part of 
this effort, the NASA Office of Education (OE) oversees education projects that span higher 
education, informal education, and precollege (K— 12) education. This portfolio augments the efforts 
of the National Science Foundation and the U.S. Department of Education, which lead the nation's 
federal STEM education efforts. 

Projects in NASA's educational portfolio address distinct but complementary objectives, which can 
be modeled as a pyramid of increasing intensity of involvement (Exhibit 1). At the base of this 
pyramid, for example, are educational outreach activities and informal education projects intended 
to appeal to the broadest population possible, to promote greater awareness of NASA's missions 
and to inspire further exploration of STEM. Another group of projects, situated in the middle tier, 
are designed to engage precollege students (and their parents) in participatory activities to deepen 
their understanding of STEM concepts and to build enthusiasm for, and persistence in, further STEM 
educational opportunities, both informal and formal. Finally, at the uppermost tiers of the pyramid 
are projects targeting those in higher education (college and graduate school) who are preparing for 
potential employment in a STEM-related field, whether in industry, academia, STEM education, or at 
NASA itself. These project may also target institutions, faculty and, to a lesser extent, K-12 
teachers. Together, the projects in this portfolio support the national agenda to maintain and 
enhance the country's expertise and leadership in STEM research and development. 


Exhibit 1 : NASA’s Educational Framework (NASA, 2007a) 
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Source: NASA 2007a, p.7 


The National Research Council (2008) recently cited the unique contribution that NASA's precollege 
programs can make to this national effort: 
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The exciting nature of NASA's mission gives particular value to projects whose primary 
goal is to inspire and engage students' interest in science and engineering, and NASA's 
education portfolio should include projects with these goals. Because engineering and 
technology development are subjects that are not well covered in K-12 curricula, 
projects aimed at inspiring and engaging students in these areas are particularly 
important. 1 

The Science, Engineering, Mathematics, and Aerospace Academy (SEMAA) project exemplifies 
NASA's efforts to inspire, engage, and educate K-12 students throughout the nation. As part of 
NASA's precollege educational portfolio, the SEMAA program is situated in the middle tier of NASA's 
educational framework, where enhancing student interest and engagement in STEM learning is of 
primary importance. Specifically, SEMAA is designed to attract and retain students through a 
progression of educational opportunities by capitalizing on the exciting nature of NASA missions and 
its expertise, technology, and resources. SEMAA, along with most of NASA's K-12 efforts, aligns with 
the national goal of engagement in STEM fields, as defined by the Academic Competiveness Council, 
to "increase students' engagement in STEM and their perception of its value to their lives." 2 

In 2008, NASA contracted with Abt Associates Inc. and its subcontractors The Urban Institute and 
the Education Development Center, Inc. (the Abt team) to design and conduct a national evaluation 
of SEMAA. This report presents the findings from this evaluation. We begin with an overview of the 
SEMAA program and its conceptual framework. Chapter Two describes the evaluation design, which 
consisted of two components— an implementation study and an outcomes study. Chapter Three 
presents the findings related to the implementation of the SEMAA program. Chapter Four describes 
the students who participated in the Saturday sessions of SEMAA, and Chapter Five presents 
findings related to outcomes associated with their participation in SEMAA. Finally, Chapter Six 
presents the conclusions and implications drawn from this evaluation. 

The SEMAA Program 

NASA's SEMAA program is a national science enrichment program, targeting students in 
kindergarten to 12 th grade and their parents (or other caregivers), that is intended to increase the 
participation and retention of students in STEM, particularly those who are historically underserved 
and underrepresented in science, technology, engineering, and mathematics fields. In the past three 
years alone, over 200,000 students, parents/adult family members and teachers have been involved 
in the SEMAA project. The goals of SEMAA are to: 

• Inspire a more diverse student population to pursue careers in STEM-related fields; 

• Engage students, parents/adult family members and teachers by incorporating emerging 
technologies; and 


1 National Research Council (2008), p. 6. 

2 U.S. Department of Education (2007), p. 18. 
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• Educate students utilizing rigorous STEM curricula, designed and implemented as only NASA 
can. 3 


In 2007, SEMAA was recognized as one of the top 18 programs in the Innovations in the American 
Government Awards competition by the Ash Institute for Democratic Governance and Innovation at 
Harvard University's John F. Kennedy School of Government. 4 


Established in 1993 at the Cuyahoga Community College in Cleveland, Ohio as a joint venture with 
the NASA Glenn Research Center, SEMAA has since expanded to other sites throughout the nation. 5 
Institutions hosting SEMAA encompass a wide range of institutions including Historically Black 
Colleges and Universities, Hispanic-serving institutions, Tribal Colleges and Universities, 
predominantly White institutions of higher education, science centers and museums, and 
elementary and secondary school districts. Since its inception, there have been 25 SEMAA sites, and 
at its peak in 2005, SEMAA was being implemented in 24 sites across the nation. Exhibit 2 presents 
the years in which sites were opened and, where appropriate, the years in which sites closed. As of 
2009, there are 14 SEMAA sites located across 12 states and the District of Columbia. Exhibit 3 maps 
the location of the current and closed sites. 


Exhibit 2: 

Number of SEMAA Sites over Time 



New Sites 

Total Number 
Open 


Year 

Opened 

Sites Closed (Original Year Opened) 

1993 

1 

1 


1997 

2 

3 


1999 

8 

11 


2001 

5 

16 


2002 

3 

19 


2003 

4 

23 


2004 

1 

23 

1 (1997) 

2005 

1 

24 


2006 


17 

3 (1999), 4 (2001) 

2007 


14 

2 (2003), 1 (2007) 

Total 

25 


11 


3 http://www.nasa.gov/offices/education/programs/national/semaa/about/index.html 

4 http://www.innovations.harvard.edu/awards.html?id=101441 

5 The NASA SEMAA Project continues to be managed by the Educational Programs Office at NASA's John H. 
Glenn Research Center at Lewis Field in Cleveland, Ohio, with assistance from contractor Paragon TEC, Inc. 
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Exhibit 3: Current and Closed SEEMA Sites 
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SEMAA sites throughout the nation rely on collaborations among partners (schools, colleges and 
universities and/or research and science centers) to provide three related components: 

• A NASA-designed, hands-on, inquiry-based curriculum aligned to national science, math, and 
technology standards at each grade level and connected to research from the NASA mission 
directorates; 

• An Aerospace Education Laboratory (AEL), featuring cutting-edge aerospace technology that 
gives students experience with aeronautical and reduced-gravity simulations; and 

• A Family Cafe to promote sustained parental involvement in their child's STEM education. 

SEMAA activities were originally designed to be implemented in three-hour sessions on Saturday 
mornings for five to eight consecutive weeks— the kindergarten to 2 nd grade sessions are five weeks, 
and the 3 rd to 12 th grade sessions are eight weeks. The AEL, an electronics and computer-equipped 
facility including a flight simulator, research-grade wind tunnel, short-wave radio receiver and hand- 
held GPS (global positioning system) for aviation, is used both to engage students with technology 
and real-world problem-solving and to train pre-service and in-service teachers. Parents are invited 
to participate in an interactive Family Cafe that includes focus groups where parents can gain an 
understanding about what their children are learning and discuss ways to support their children's 
engagement in science and mathematics learning both in and outside of school; family nights, where 
parents and students come together to work on hands-on STEM-related projects; and home-based 
STEM-linked activities for students and adult family members to share (National Research Council, 
2008). Trained facilitators and invited speakers engage parents in a variety of activities, from 
demonstrations of science-related experiments, to hands-on modeling projects, to interactive 
discussions of STEM topics. For each grade level, SEMAA offers a distinct, age-appropriate set of 
curricular enhancements, with an emphasis on hands-on projects and exposure to current 
technology. 

The Saturday session model was designed with up to three eight-week sessions offered each 
academic year, in addition to a one-week session offered during the summer to help students 
transition from the previous grade level of SEMAA activities to those anticipated for the following 
grade level. In a given academic year, a student might enroll in an academic session and a summer 
session. Students may enroll at any grade level as there are no prerequisites for participation in any 
grade. 

Although the Saturday model is the original and most common SEMAA model, some sites offer an 
after-school or in-school program instead. In-school SEMAA models typically expand the SEMAA 
curriculum for integration into a school's existing science curriculum. After-school models are 
sometimes used in areas where families have difficulty attending on Saturdays (e.g., because of 
transportation barriers). The implementation of SEMAA is discussed further in Chapter Three. 
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A National Evaluation of SEMAA 


Consistent with the American Competitiveness Council's recommendations for rigorous, 
independent evaluations of federal STEM program and fostering knowledge of effective practices 
through improved evaluations (U.S. Department of Education, 2007), the National Research Council 
(2008) recommended that NASA develop a coherent plan for evaluating its K-12 STEM education 
programs and use the evaluation findings to inform program implementation and design. The NRC 
made specific recommendations for an evaluation of NASA's SEMAA project, recommending that 
the evaluation of SEMAA be based in the program's theory of change, that the evaluation include an 
appropriate comparison group, and that an external evaluator conduct the evaluation. To evaluate 
SEMAA, the Abt team designed a comprehensive, mixed-methods evaluation with two key modules 
to study the implementation and impact of SEMAA. The design and implementation of the impact 
and process study modules were led by Abt Associates and The Urban Institute, respectively. 

The impact evaluation component was designed as a multi-site, randomized control trial (RCT) to 
determine the short-term impacts of SEMAA on K-12 students and their families. It investigated the 
impact of SEMAA on short-term outcomes such as student interest and efficacy in STEM, student 
engagement in STEM activities, and families' engagement in students' STEM-related activities. The 
process evaluation component utilized document review and telephone interviews with the SEMAA 
sites' project directors to investigate variation in how sites implement SEMAA and contextual factors 
important in implementation. 

This evaluation examined the implementation of SEMAA at the various sites, and found that three 
models prevail— the original Saturday model, an after-school model, and an in-school model- 
defined by when sites offer the SEMAA components. The impact study was conducted with 
participants in the Saturday model only, and the report includes discussion of how the findings from 
the impact study are particularly relevant given the pattern of model selection across sites. In 
addition, findings from the current evaluation can inform the implementation of SEMAA in 
continuing sites, as well as future evaluations that may explore the long-term outcomes of SEMAA. 
While short-term outcomes are important in understanding how the SEMAA program may affect 
participants, a longitudinal evaluation of SEMAA would be necessary to understand the longer-term 
impacts of the program. NASA has begun to develop and implement data tracking tools that will 
track SEMAA participants— as well as participants in other NASA educational programs— over time, 
which will facilitate any future efforts to investigate the long-term outcomes of its programs. 

The Issue and Related Factors 

The STEM workforce depends on an educational pipeline that prepares students for postsecondary 
STEM education and retains those students to degree completion. Unfortunately, the pipeline into 
and through college-level STEM preparation is weak, and has been described as "leaky," particularly 
for women, African-Americans, and Hispanics— as they progress to successively higher levels of 
education, members of these groups increasingly opt out of STEM fields (e.g., Oakes, 1990; Watt, 
Eccles, and Durik, 2006). Between 1995 and 2001, less than one-quarter of all post-secondary 


6 


National Evaluation of SEMAA 


Abt Associates Inc. 


students had— at one time— declared a major in a STEM field, and by 2001, only 27 percent of these 
students had earned a bachelor's degree in a STEM field. Women were less than half as likely as 
men to declare a STEM major, but once having done so, they were equally likely as men to earn a 
STEM bachelor's degree. Asian/Pacific Islander students were almost twice as likely as members of 
other racial and ethnic groups (Whites, Blacks, Hispanics) to declare a STEM college major. Among 
declared STEM majors, Whites and Asians were more likely than Blacks, Hispanics, or others to 
complete a STEM bachelor's degree (NCES, 2009a). 

SEMAA is designed to increase student participation and retention in STEM, particularly students 
from groups traditionally underrepresented in STEM fields. There is a growing body of research that 
links the intermediary outcomes of SEMAA— such as increasing student involvement in science 
(including course taking) and affective outcomes— to important STEM educational and career 
outcomes. In addition, there is some preliminary evidence that engagement in STEM enrichment 
programs, such as SEMAA, is related to these intermediary outcomes. Below we discuss this 
literature, and follow with a presentation of SEMAA's program theory, which articulates how SEMAA 
is intended to achieve its desired impacts. 

The importance of high school course-taking 

Evidence points strongly to the importance of adequate preparation in high school science and 
mathematics for more advanced, college-level work in these fields. Course enrollment decisions in 
high school, especially decisions about the number and rigor of mathematics courses, are strongly 
related to college and earnings outcomes (e.g., Levine and Zimmerman, in press; Rose and Betts, 
2004). Taking more math and science courses in high school was associated with an increased 
likelihood of choosing a technical college major even when prior 8 th grade achievement, interest in 
math and science, and differences in state graduation requirements (number of years of 
mathematics required) are taken into account (Federman, 2007). Mathematics preparation, in 
particular, is a gateway to many other fields of study; students who complete rigorous mathematics 
courses in high school outperform their peers in college-level physics, chemistry, and biology (Sells, 
1980; Sadler and Tai, 2007). 

Although enrollments in high school science and mathematics courses improved nationally between 
1982 and 2004, in terms of both number of courses completed and the level of the highest courses 
completed, by 2004 only 18 percent of high school graduates had completed at least one advanced 
science course (physics II, chemistry II, and/or advanced biology) and only 14 percent had completed 
a calculus-level mathematics course (NCES, 2007). Advanced course-taking among girls was not 
significantly different from that of boys, but racial disparities in the most advanced levels of math 
and science course-taking persisted: although nearly 40 percent of Asian/Pacific Islander and 20 
percent of White students had completed at least one advanced science course (physics II, 
chemistry II, and/or advanced biology), only 11 percent of Black students, 9 percent of Hispanic 
students and 7 percent of American Indian students had completed such coursework. Data show a 
similar pattern of racial disparities in completion of at least one calculus-level mathematics course 
(NCES, 2007). 
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Even as educators and policy-makers target improvements in high school courses and completion 
rates, research suggests that there are additional factors impeding STEM persistence. Although 
adequate achievement is needed to continue to progress through successively more advanced levels 
of STEM education, achievement alone does not account for persistence in STEM learning or career 
aspirations. For example, despite the fact that girls complete high school math and science courses 
at similar rates and with similar levels of achievement as boys, women are half as likely as men to 
ever declare a college major in a STEM field (NCES, 2009a). If actual achievement in a subject were 
equally predictive of subsequent enrollment in that subject for boys and girls, then girls should be 
just as likely, in the absence of other mediating factors, to pursue a college major in a STEM 
discipline. 

As described below, research suggests that students' interests, the value they place on STEM 
learning, their own and their parents' perception of their ability, and their parents' educational 
expectations also play critical roles in STEM educational outcomes. Moreover, these types of 
affective factors— interests, perceived ability, values— may be particularly important levers to 
encourage STEM persistence for members of groups traditionally underrepresented in STEM fields. 

Affective factors influencing STEM persistence: Interest and self-efficacy 

Interest in school subjects or activities related to such subjects is positively correlated with 
enrollment and achievement in those subjects. A meta-analysis of research on interest-achievement 
relationships for mathematics, physics, and science in general found statistically significant average 
correlations of .31 or higher; for biology, the average correlation was .16 but still significantly 
positive (Schiefele, Krapp, and Winteler, 1992). Longitudinal studies of student interest and 
achievement in science and math support this evidence (e.g., Shernoff and Hoogstra, 2001; Watt, 
Eccles, and Durik, 2006). For example, in one study, researchers compared students' interest, 
enjoyment, and level of concentration during high school science and math courses to their choice 
of college major and college grades two years later. Controlling for high school math and science 
grades, students' prior interest, enjoyment, and level of concentration during high school science 
classes positively predicted selection of a science major in college. Moreover, these same affective 
factors predicted overall college academic performance, even controlling for prior high school 
performance in math and science (Shernoff and Hoogstra, 2001). 

Interest and enjoyment of science may be particularly important for girls' desire to take science 
courses. In a study of high-ability 4 th to 6 th grade girls and boys enrolled in a gifted pull-out program, 
which used the Test Of Science Related Attitudes (TOSRA; Fraser, 1981), the number of science 
courses girls selected was significantly correlated with their reported enjoyment of science lessons, 
science leisure activities, and their perception of scientists as "normal"; no such significant 
relationships were found for boys in the study (Ferenga and Joyce, 1998). 

Although interest in math and science may support continued engagement and achievement, much 
more research has focused on the role of self-efficacy, namely, a person's expectation that they 
have the necessary abilities to carry out a task, in predicting academic and career outcomes in 
general and STEM outcomes in particular (Bandura, 1993). Self-efficacy is consistently found to 
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correlate with course enrollment and career aspirations (Bandura, Barbaranelli, Caprara, and 
Pastorelli, 2001; Betz and Hackett, 1983; Lent, Brown, and Larkin, 1986; Simpkins, Davis-Kean, and 
Eccles, 2006; Watt, Eccles, and Durik, 2006). Moreover, self-efficacy for science and math may 
predict continued interest in these fields. For the predominantly White students from high 
socioeconomic status (SES) households in the Michigan Childhood and Beyond study, self-concept of 
science and math ability in 6 th grade was positively correlated with their subsequent interest in 
these subjects in 10 th grade (Simpkins, Davis-Kean, and Eccles, 2006). In the nationally 
representative National Educational Longitudinal Study of 1988 (NELS: 88), self-efficacy predicted 
sustained interest in a STEM career more strongly than did academic proficiency (Mau, 2003). 

Self-efficacy emerges early in development as a predictor of subsequent educational choices and 
occupational aspirations. Children who believed, in 6 th grade, that they were skilled in math or 
science subsequently completed more science and math courses in high school, even controlling for 
parents' educational and income levels and for students' course grades in 5 th and 10 th grade 
(Simpkins, Davis-Kean, and Eccles, 2006). In a study with a socio-economically diverse sample of 
children 11 to 15 years of age, students' self-efficacy for occupational functions hinged on their self- 
efficacy for related academic coursework but not on their academic performance in related courses. 
Children who perceived themselves as capable of performing scientific and technological job 
functions favored professorial (i.e., academic), creative occupations as well as occupations requiring 
technical but less cognitively demanding skills (e.g., mechanical manufacturing or repair) over 
caretaking, educational or social service careers (Bandura et al., 2001). Gender disparities were 
found, with boys having stronger self-efficacy than girls for coursework in math and geography, and 
for science and technological occupational functions. 

Although gender disparities in self-efficacy may emerge early and shape career aspirations (e.g., 
Bandura et al., 2001), their effect on actual STEM educational choices may depend on the context in 
which students make such choices. For example, although girls in the Michigan Childhood and 
Beyond Study had a lower perception of their math skill than boys, independent of their prior math 
achievement, a difference that persisted from 6 th to 10 th grade (Simpkins, et al., 2006), they did not 
complete fewer numbers of math or science high school courses. Girls were, however, less likely to 
engage in math activities outside of school than boys. Simpkins et al. (2006) suggest that the high 
school students in their sample may have been more restricted in opting out of mathematics and 
science, due to their own and their parents' college aspirations. Once enrolled in college, however, 
where they can exercise more control over educational decisions, young women are less likely than 
young men to declare a science or mathematics major (NCES, 2009a). Support for this notion comes 
from data on a similar sample of Australian high school students, where the educational system 
allows students to choose, after 10 th grade, the difficulty level of their mathematics courses in 11 th 
and 12 th grade. A smaller proportion of girls than boys in this sample chose the highest difficulty 
courses, and a greater proportion of girls than boys opted for the lowest difficulty levels of 
mathematics (Watt et al., 2006), despite no difference in grade 9 or grade 11 mathematical 
achievement. 
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The role of parents in children's STEM outcomes 


Among the most ubiquitous findings of the influence of family on children's academic development 
is the association between children's educational outcomes and their parents' SES and educational 
attainment. Much of this evidence comes from studies of language and literacy development, in 
which a large literature shows that more highly educated, higher income parents both read and talk 
to their children more often, about a broader range of topics and using more complex sentence 
structure than other parents (e.g., Hart and Risley, 1995). These parents also have higher aspirations 
for their children's education, may be more likely to provide children with supplemental educational 
experiences outside of school (e.g., music lessons, academic tutoring, trips to science museums), 
and may be more likely to model engagement in intellectual pursuits (e.g., reading, non- 
occupational hobbies) in the home (Eccles, 2005). Longitudinal data support some of these 
hypothesized links. For example, data from the Child Development Supplement (CDS) to the Panel 
Study of Income Dynamics (PSID) illustrate how educational attainment of parents and their children 
are linked: more highly educated parents have higher expectations for their children's education; 
parents with these higher expectations invest more time at home in their children's education; and 
the more of this time parents invest, the higher children score on standardized achievement tests 
(Davis-Kean, 2005; cited in Eccles, 2005). 

In addition, parental behaviors to encourage engagement in mathematics are associated with 
children's enduring interest and engagement in science and math over time. Researchers in one 
study measured students' interest in math in early elementary school and six years later after they 
transitioned to middle school, and found that children's interest in math was related to mothers' 
(but not fathers') purchase of math and science toys; further, the frequency of parents' participation 
in math and science activities with their children was positively related to children's interest in math 
six years later (Jacobs and Bleeker, 2004), controlling for gender, children's prior interest and 
parents' conception of their child's math ability. 

A growing body of research suggests a more precise model of the way in which parental 
expectations and behaviors affect children's achievement. Rather than directly affecting math or 
science achievement, parental expectations for their child's engagement in math and science 
learning, encouragement and shared involvement in math and science activities influence children's 
self-efficacy for math and science; the more parents signal their belief in the child's ability, the 
greater the child's own self-efficacy, and in turn, children's higher self-efficacy is associated with 
stronger achievement. Although children's self-efficacy for mathematics declines from 1 st to 12 th 
grade, the rate of this decline is less steep for children whose parents have confidence in their 
child's mathematics ability, even when holding constant both children's actual math performance 
and teachers' opinions about these students' math abilities (Frederick and Eccles, 2002; cited in 
Eccles, 2005). 

Corroborating evidence about the effects of parental ratings of their children's abilities and 
children's own self-efficacy and STEM outcomes comes from other longitudinal studies (e.g., 

Bandura et al., 2001; Bleeker and Jacobs, 2004). Mothers' perceptions of their children's 
mathematics ability, measured when children were in 6 th grade, was significantly related to 
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children's self-concept of their math ability measured four years later when these children had 
reached 10 th grade— this relationship was independent of children's actual math ability in 6 th grade 
(Bleeker and Jacobs, 2004). Moreover, these same mothers' predictions of the likelihood of their 7 th 
graders' success in a math-related career significantly predicted their children's career self-efficacy 
in early adulthood (age 19-20) and career choices made in early adulthood (age 24-25), again 
independently of children's actual math ability. Among the young adult outcomes, mothers' 
predicted likelihood of their children's success in a math-related career was positively related to 
young adults actually working in a physical science or computing career versus a non-science career, 
regardless of their college attendance (Bleeker and Jacobs, 2004). 

Out-of-school STEM enrichment activities 

Parents, educators, and policy-makers concerned with children's educational and economic 
outcomes have long looked for ways to supplement students' formal schooling with additional 
learning opportunities. A wide variety of public and private after-school, weekend, and summer 
programs targeting elementary and secondary students are now available to many families. As 
concern about STEM education has grown, the number of these programs providing students with 
math and science enrichment has risen. 

Out-of-school science and math activities have the potential not only to reinforce students' content 
knowledge, but also to enhance their interest, improve their sense of self-efficacy for science and 
math-related tasks, and illustrate the relevance of science and math in their daily lives and for their 
career aspirations. In addition, out-of-school activities can provide parents an opportunity to jointly 
engage in science or math activities with their children; through this shared engagement, parents 
may signal that they see math and science as important endeavors and that they see their children 
as competent agents in these fields. 

Increasingly, policy-makers seek evidence for the effectiveness of these programs. Non- 
experimental evaluations of the effectiveness of such "out-of-school-time" (OST) programs have 
suggested a host of positive outcomes for participants in such programs, including improved 
achievement and stronger interest in, and more positive attitudes towards science and 
mathematics. For example, an evaluation of 96 projects run by The After School Corporation (TASC) 
over a four-year period showed that math achievement scores for 3 rd to 8 th grade participants rose, 
on average, after two years (Reisner, White, Russell, and Birmingham, 2004; cited in Levine and 
Zimmerman, in press). A meta-analysis of 33 studies of OST math enrichment programs suggests 
that low-achieving participants in such programs experience positive and statistically significant 
gains in achievement (Lauer et al., 2004). Notably, however, both the TASC study and most of those 
in Lauer's (2004) meta-analysis were non-experimental evaluations that did not include an 
appropriate comparison or control group. 

In light of prior research (e.g. Mau, 2003; Shernoff and Hoogstra, 2001; Simpkins et al., 2006) 
suggesting that improved STEM achievement in elementary and secondary school alone is 
insufficient to produce long-lasting, persistent engagement in STEM, it is important to examine the 
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effects of STEM enrichment programs on student interest, self-efficacy, and motivation to pursue 
additional STEM education. 

Many participants in STEM enrichment program report positive benefits of these programs, 
including increased interest in and self-efficacy for science, and enhanced interest in pursuing a 
scientific career (e.g., Fadigan and Hammrich, 2004; Markowitz, 2004; Stakes and Mares, 2005). 

Such assertions must be treated with caution. Although the majority of past participants in the 
Women in Natural Sciences (WINS) program perceived the program as having influenced their 
educational and career trajectories, what actually differentiated those who eventually pursued or 
did not pursue a STEM career was the number of honors or advanced placement science and math 
high school courses completed; total number of math and science courses, combined SAT scores, 
type of school attended, number of out-of-school-time activities and— notably— years in the WINS 
program were not associated with girls' subsequent career paths (Fadigan and Flammrich, 2004). 

More rigorous studies of the effects of STEM enrichment programs are less common. In one random 
assignment study, applicants to a summer inquiry-based science camp for rising 7 th and 8 th graders 
were randomly assigned to participation or nonparticipation (the control group). Both groups 
showed a decline in interest two years after participation, but program participants showed smaller 
declines than those in the control group. Individuals who had not applied to the program were also 
followed, and they had lower interest at baseline and showed a decline in interest from middle to 
high school (Gibson and Chase, 2002). 

Other studies also suggest that the incorporation of such inquiry-based or field-based experiential 
science or math experiences may increase interest in science (e.g., Barnett et al., 2006; Sikes and 
Schwartz-Bloom, 2009). Field-based methods, even when students may not formulate their own 
research questions, may have benefits for students' interest in science. In a university-school district 
partnership to expose urban high school students to a hands-on, field-based urban ecology program, 
researchers found suggestive evidence that such experiences may prevent declining interest in 
science over the course of an academic year (Barnett et al., 2006). 

The SEMAA project is grounded on these ideas that emerge from the existing literature. The 
program's theory of change and how this informed the current study are described in the next 
section. 

Conceptual Framework 

SEMAA's theory of change 

To increase the number and diversity of students pursuing STEM education and careers, SEMAA is 
designed to maintain and increase student interest in STEM activities and motivation to persist in 
these activities. SEMAA seeks to build interest and motivation by engaging students and their 
families in hands-on, participatory activities that align with formal educational standards in science 
and mathematics and enhance learning. The original SEMAA model was designed so that these 
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activities would take place outside of school on Saturday mornings. As described in greater detail in 
Chapter Three, sites are also implementing models wherein the SEMAA sessions are conducted after 
school or within the school day. 

The SEMAA model posits that effective engagement goes hand-in-hand with fostering a positive 
attitude about STEM activities. The more students engage with the content, materials, and 
processes of STEM inquiry in an informal, supportive context, the more positively they will feel 
about STEM activities and their role in such inquiry; and vice versa, as students' self-efficacy 
improves, the more they will be motivated to further engage in STEM activities and learning. In 
addition, the more their parents share an interest in STEM activities, and the more support for 
education parents can provide students, the more likely it is that students will participate in 
additional STEM activities, and enroll and persist in school-based science and mathematics courses. 

The SEMAA model reflects an implicit theory of how to effect change in student behavior. According 
to the theory of change, engaging students means providing them opportunities to participate 
actively in STEM tasks and enlisting the support of their parents. Effective engagement builds 
interest and motivation in STEM and enhances STEM learning. This greater motivation and 
understanding increases the desire for further engagement, increases students' involvement in 
STEM activities, and improves the likelihood that they will persist in STEM learning, both informal 
and formal. Ultimately, this persistence will lead to the more desired long-term outcomes— 
increased enrollment and achievement in STEM coursework, more undergraduates interested in and 
adequately prepared to major in STEM disciplines, and a greater flow of individuals into STEM- 
related careers. 

Exhibit 4 models SEMAA's theory of change. Moving from left to right, the model makes explicit the 
links between the project inputs— those elements necessary to enact the project at a site— through 
the activities that SEMAA sites conduct, to the outputs of these activities, to intended short-term 
and long-term outcomes that SEMAA is designed to effect ultimately. Each of these elements in the 
logic model is described below. 
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Exhibit 4: Logic Model Depicting SEMAA’s Theory of Change 


SEMAA SEMAA SEMAA SEMAA 

Program Inputs Activities Outputs Outcomes 


NASA provides: 

• Funding support to projects 

• Aerospace Educational 
Laboratory (AEL) equipment, 
installation and training 

• Curriculum enhancement 
activities (CEAs) aligned to 
national math, science and 
technology standards 

• Training to local SEMAA staff 


Host sites provide 

• Classroom and 
facilities/laboratory space, 

• Site director, teachers, 
family cafe coordinator 

• Administrative support, 
office supplies 

• Staff to maintain and 
operate the AEL 

• Potential partner 
organizations for matching 
funds, community 
involvement 


Recruit students & parents from 
underserved populations 

During the academic year, offer 
grade-level classes (fall, winter, 
and spring sessions): 

• K-2 nd grades meet once/week 
for 5 weeks 

• 3 rd -12 th grades meet once/week 
for 8 weeks 

During summer, offer one-week 
"bridge" sessions (3 hours/day) 
for each grade level targeted 

Incorporate "Hands-on/Minds- 
on" activities in classes 

Facilitate student use of AEL for 

collaborative, real-world 
application of STEM concepts 

Facilitate Family Cafe sessions for 
parents/families 

Collect feedback from participants 
to inform program 
improvements 

Develop a sustainability plan and 
form partnerships with local 
community, industry, academic 
partners 


Students: 

• Engage in inquiry-based 
educational activities for up 
to 21 hours (K-2 nd grade) or 
36 hours (3-12 th grade) per 
year 

• Experience different roles 
and tasks related to STEM 
learning in SEMAA 
classrooms and in the AEL 

Parents/families: 

• Engage in discussions of 
their role in supporting 
child's education 

• Learn about educational 
resources available from 
NASA and local sources 

Sites: 

• Report data on enrollment, 
attendance, participant 
satisfaction and other 
outcomes 

Partnering organizations: 

• Provide matching funds and 

resources; 

• Supply guest speakers and 

offer site visits to research 
and development facilities 


For students: 

• More positive attitudes 
about STEM 

• Improved self-efficacy for 
STEM learning 

• Greater participation in 
STEM activities outside of 
school 

• Greater interest in formal 
STEM educational 
opportunities 

• Greater interest in careers in 
STEM-related fields 

• Continued enrollment in 
SEMAA classes through a 
progression of grade levels 

For parents/families: 

• Increased support for their 
child's engagement in STEM 
activities and formal STEM 
educational opportunities 

• Increased value placed on 
child becoming proficient in 
science, mathematics 

Self-sustaining SEMAA sites 


Notes: SEMAA projects must be operated by 2- or 4-year HBCUs, HSIs, Tribal Colleges & Universities, & Other Minority Universities 


SEMAA 

Impacts 

• Increased enrollment in 
high school STEM courses 

• Improved student 
achievement in STEM 

• Increased diversity and 
number of post-secondary 
graduates of with STEM 
degrees 

• Increase in number and 
diversity of those pursuing 
careers in STEM-related 
fields 

• Increase in public 
engagement and support 
for STEM research and 
development in general, 
and for NASA's mission in 
particular 


SEMAA activities 

Once they have secured funding and received approval for their standard operating procedures 
(negotiated with NASA's National SEMAA Office), sites then recruit students and their families from 
area school districts to enroll in classes. 6 SEMAA deliberately targets populations typically 
underserved and underrepresented in STEM disciplines— local projects are often located at 
minority-serving institutions of higher education. To maximize enrollment, classes are offered when 
students and parents are most likely able to participate. In many sites classes take place for three 
hours on Saturday mornings, with Family Cafes running concurrently, but some sites offer courses as 
after-school extracurricular activities, while still others offer an in-school model in which the 
curriculum enhancement activities are incorporated school-wide into the existing science or math 
curriculum. 

The curricular enhancement activities are designed to encourage persistence in SEMAA throughout 
grades K-12. At each grade level, they are developmentally appropriate, and across grade levels, 
activities are somewhat cumulative so that students learn new concepts and engage in novel STEM 
activities. In addition, SEMAA projects use the AEL to expose students in secondary grades (grades 6 
to 12) to real-world applications of up-to-date technologies so that students see how STEM content 
is actually used to solve problems like those NASA mission directorates encounter (e.g., how 
microgravity in space presents challenges to be solved, how to design aircraft, how to navigate 
aeronautical and aerospace vehicles over the large distances involved using GPS technologies, how 
to coordinate communications, etc.). 

Parent participation in SEMAA activities is optional. SEMAA offers parents (or other caregivers) a 
chance to experience some of the activities their children are engaged in through family nights, field 
trips to science museums, and showing parents STEM activities they can do together at home. In 
addition, at Family Cafe sessions a SEMAA family coordinator organizes guest lectures by STEM 
professionals and educational providers and facilitates discussions with parents about how to 
support their children's education, and solicits parents' concerns about their children's education 
and development. Parents receive information about other NASA educational opportunities and 
other resources available to families in the local community. 

Beginning in the first year of implementation, each SEMAA site also actively develops a sustainability 
plan, including partnerships with school districts, local industries, non-profit and community youth 
groups, and colleges and universities. These partnerships are necessary to ensure that the site can 
continue to operate after the end of its NASA grant term. Recruitment of families and solicitation of 
partners work together to build a broad base of stakeholders interested in the continuance of the 
local SEMAA site. 


Note that sites are not necessarily required to implement classes for all grade levels (K-12). Sites may 
choose to target course offerings to a smaller range of grades (e.g., grades 6-8), and may expand to 
include additional grades as student and parent interest, resources, and space allows. 
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Finally, sites collect data on a range of indicators to inform program improvements and track 
progress. Such data include enrollment, attendance at classes, number and types of NASA materials 
used, topics covered, and feedback from participating students, parents, and teachers, including 
satisfaction with project offerings and interest in STEM careers. These data were input to the NASA 
Education Evaluation Information System (NEEIS) database and aggregated to assist sites and NASA 
in monitoring SEMAA's implementation and outcomes. 7 

SEMAA outputs 

Each year, SEMAA should have provided 3 rd to 12 th grade students up to 36 hours of exposure to 
SEMAA content (3 hours per week for 7 weeks of an 8-week academic year session, 8 plus 15 hours 
during a 1-week summer session), or up to 21 hours of exposure for students in kindergarten to 2 nd 
grade. Parents should have had the opportunity to participate in discussions of their role in 
supporting their child's education and to learn about resources available to encourage and assist 
their child's engagement in STEM learning (e.g., school or community mentoring/tutoring, summer 
science camps, resources for low-income families to participate in STEM activities). At the end of 
each SEMAA session, staff should have accumulated administrative and participant feedback data 
and entered these data into NEEIS for reporting and monitoring purposes. Finally, as a result of staff 
outreach, the site should have established partnerships with local industries and agencies who have 
given and pledged continued support in the form of matching funds, guest speakers, or hosting visits 
by students and parents to their facilities. 

SEMAA short-term outcomes 

If SEMAA is successful, then its activities and outputs should result in the anticipated outcomes for 
students, parents, and the sites themselves. Students should feel more positive about engaging in 
STEM activities and should increasingly view themselves as capable of engaging in such activities; 
they should be more inclined to engage in STEM educational opportunities, both in and out of 
school; they should express greater interest in the possibility of pursuing a STEM career; and they 
should be more likely to enroll in future SEMAA sessions and in formal STEM coursework. As a result 
of their exposure to SEMAA, parents should likewise have a more positive view of STEM, particularly 
of the importance of STEM education for their children and their own role in supporting their child's 
learning; also, they should be more likely to engage in STEM activities with their child and encourage 
their child to pursue STEM educational opportunities in and outside of school. SEMAA sites should 
see direct benefits of student and parental participation and from their efforts to secure 
partnerships; enrollments in subsequent sessions should increase; and external stakeholders should 
be increasingly responsible for sustaining SEMAA long term. 

SEMAA long-term outcomes 

The final premise underlying the SEMAA logic model is that the outcomes for precollege students 
and their parents— increased student interest in STEM activities and improved self-efficacy, 
increased participation in out-of-school STEM activities, greater interest and enrollment in formal 

7 NASA is currently implementing a new online data monitoring system to replace NEEIS. 

8 The final week of the academic session does not introduce additional content. 
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STEM coursework, increased parental engagement in STEM activities with their child and stronger 
support for their child to enroll in formal STEM coursework— will result collectively in subsequent 
impacts on the college-level STEM pipeline and subsequent STEM labor force. That is, if SEMAA 
helps increase precollege students' interest and participation in informal and formal STEM activities, 
a greater proportion of such students should persist through high school STEM coursework that 
prepares them for college-level STEM courses. In turn, a greater proportion of students should 
pursue and complete undergraduate degrees in STEM disciplines. With more students trained in 
college-level STEM, a greater proportion will enter the STEM labor force, either directly or via 
graduate level education. These impacts will help meet the national need for more individuals, and a 
greater diversity of individuals, pursuing STEM careers and thus contributing to the research and 
development needed for the U.S. to remain a global leader in such fields. 

How the SEMAA program theory informed the evaluation 

The design of the national evaluation of SEMAA was informed by SEMAA's program theory. 

Although NASA provides sites with materials and guidance to implement the SEMAA project— 
originally envisioned as a Saturday program— there is variation in how sites implement components 
in response to the particular needs of their student population, characteristics of the host and 
partner institutions, and resource availability. The process study focused on this variation, and its 
implications for successful implementation of SEMAA, with the specific aim of uncovering 
alternative models of implementation of the SEMAA project and shedding light on issues 
surrounding sustainability. 

The outcomes of the impact evaluation follow directly from the SEMAA project logic model. The 
evaluation design deliberately targeted the short-term outcomes anticipated in the SEMAA program 
theory (shown in the fourth panel of Exhibit 4)— including student attitudes, engagement in STEM 
activities (both school-based extracurricular and out-of-school activities), interest in science, specific 
STEM courses, college majors, and STEM careers, and parental engagement in STEM activities and 
support for their child's enrollment in formal STEM courses. Although previous evaluations of 
SEMAA had examined some of these same outcomes, they had not provided rigorous evidence that 
SEMAA affected these anticipated outcomes. For example, measures were administered only post- 
participation (i.e., no baseline measures of these outcomes were collected prior to participation in 
SEMAA) and only to SEMAA participants (there was no comparison group of nonparticipants). 9 
Because the SEMAA program theory anticipates that these short-term outcomes are precursors to 
longer-term outcomes, it was important to first implement a rigorous design to look for the 
intended effects of SEMAA on these more proximal outcomes. In addition, the program theory 
suggests that cumulative and persistent exposure to SEMAA may enhance STEM knowledge and 
improve learning. Therefore, we explored whether the effects of the program differed for 
individuals who had previously participated and those who were participating for the first time. 


Previous evaluations were either internal evaluations or conducted by the contractor operating the 
National SEMAA Office (e.g., Bondurant, Davis, Davey and Davis, 2008; NASA, 2007b) 
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To measure SEMAA's impact on anticipated outcomes, the evaluation implemented a randomized 
control trial, in which applicants to SEMAA— those with similar levels of interest in SEMAA 
participation— were randomly assigned to a treatment (offer of enrollment in SEMAA) or a control 
group (delayed offer to enroll in SEMAA after the conclusion of the study). Both baseline and follow- 
up data were collected from each group to provide a rigorous comparison of differences in 
outcomes. Because the design precluded direct measures of student behaviors, such as actual 
engagement in a range of out-of-school STEM-related activities or actual enrollment in STEM 
courses, the measures focused on self-reported interest and frequency of engagement in such 
activities and courses. In addition, because the program theory predicted that participation in 
SEMAA would result in more positive attitudes about science and other STEM tasks, the survey 
incorporated measures of self-efficacy (anxiety about science, self-confidence in science). 

Because a key component of SEMAA is the engagement of parents (e.g., via the Family Cafe) in 
STEM activities and in their children's education, parent outcomes were also measured. These 
included parents' support for their children's out-of-school science activities, the likelihood that 
parents would encourage their children to enroll in particular high school and college-level STEM 
courses, and parents' perception of the importance that their children reach proficiency in science 
and mathematics. By exploring the short-term outcomes, the results of this evaluation lay the 
foundation for exploring the effect of SEMAA on the long-term impacts shown in the final panel of 
Exhibit 4. 
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Chapter 2. Study Design 


The evaluation was designed with two components, an implementation study and an impact study. 
The implementation study documented the variation in implementation of SEMAA across sites and 
the impact study assessed the effects of participation in the SEMAA program on short-term 
outcomes for student participants and their families. 

Research Questions 

The implementation and impacts components of the evaluation were designed to address the 
following research questions: 

Implementation Study 

• What are the main models of implementation represented among SEMAA grantees? 

• How do contextual factors affect implementation of programs? 

• How have individual sites used partnerships with other entities to implement the program? 
Impact Study 

What impact does SEMAA have on students and parents? Specifically: 

• What is the effect of SEMAA on students' engagement, interest, and self-efficacy in STEM 
activities? 

• What is the effect of SEMAA on students' interest in pursuing STEM coursework and career 
opportunities? 

• What are the impacts of the program on first-time SEMAA participants? On returning 
SEMAA participants? 

• Does SEMAA enhance parental support for their child's engagement in STEM learning? 

Implementation Study Methodology 

The implementation evaluation capitalized on existing data about local site implementation from 
extant sources. To fill gaps in the extant data, structured interviews with site directors at each of the 
14 active sites were conducted. While the main programmatic goal of the SEMAA project is to 
succeed in engaging and educating students, another goal is to graduate successfully from NASA 
funding, i.e., to become a self-sustaining project. In light of the latter goal and the presence of sites 
that no longer offer the program, we included an analysis of efforts towards institutionalization or 
sustainability at both open and closed sites. 

Documents reviewed included standard operating procedures (SOPs) that provide programmatic 
details of each site, quarterly performance reports, and project websites. This review became the 
basis of summaries of key information across sites and guided the development of the interview 
protocol (included in Appendix B). Interviews were conducted with all open sites (14) and with 6 of 
the 11 closed sites. (See Appendix A for a full list of site interviews conducted, representing a 100 
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percent response rate for active sites and a 55 percent response rate for closed sites.) The project 
directors for the remaining five closed sites did not refuse participation; rather, they were no longer 
at the institutions that had hosted the SEMAA projects and, in spite of the use of tracking services, 
could not be located to interview. 

The telephone interviews with active site directors provided implementation data on such topics as 
project goals and history, project management and governance, project components and activities, 
student participation, teacher training, partnerships and collaborations, and facilitating and 
inhibiting factors in implementation. Interviews with closed sites, which were designed shorter to 
encourage participation, focused on questions about sustainability and problems encountered. 
Telephone interviews were conducted between June and September 2009. Additional details about 
the interviews can be found in Appendix A. 

Impact Study Methodology 

The impact study was designed as a randomized control trial (RCT) of students in 4 th through 8 th 
grades (and their parents). The study was conducted across six SEMAA sites that implemented the 
Saturday model in Fall 2009. 10 In each site, student applicants were randomly assigned to a 
treatment (participation in an eight-week SEMAA curricular enhancement program in the Fall of 
2009) or control (not participating in the Fall 2009 session) condition. * 11 Student and parent pre- and 
post-test surveys were administered to each group to measure both student outcomes (e.g., student 
interest and engagement in school-based and out-of-school STEM related activities; self-efficacy in 
science and mathematics; interest in STEM coursework and careers) and parent outcomes (e.g., co- 
participation with children in STEM-based activities; perception of their child's ability and 
performance in science and mathematics; support for their child's interest in pursuing STEM 
coursework and/or careers). Program effects were measured by comparing the average outcomes 
for the treatment group and comparison group at follow-up, controlling for baseline measures and 
demographic characteristics of students and parents. 

The impact study required certain modification to sites' recruitment and application processes. 
Specifically, in addition to the standard SEMAA application, the application packet included baseline 
surveys for students and parents to complete. Also, additional lead time was necessary to randomly 
assign applicants into treatment and control groups for the study, thus necessitating that all 
recruitment occur during the summer and prior to the school year. 

Sources of data 

To measure short-term outcomes, the study team developed baseline and follow-up student and 
parent surveys for administration to treatment and control groups. To measure student engagement 

10 One site was dropped from the study due to insufficient number of applicants for the Fall 2009 session. 

11 These individuals were eligible to participate in SEMAA in the Winter or Spring 2010 session, after data 
collection for the evaluation ended. 
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in science activities and interest and self-efficacy in science, survey items from two existing 
instruments— the Modified Attitudes Towards Science Inventory (mATSI; see Weinburgh and Steele, 
2000) and the Math and Science Interest Survey (Hulett, Williams, Twitty, Turner, Salamo, and 
Hobson, 2004)— were modified (as needed) and incorporated into the study surveys. Remaining 
items were developed, pilot-tested, and revised by the study team. Baseline and follow-up surveys 
for students and parents appear in Appendix B. 

The student and parent surveys were administered twice to both the treatment and control groups. 
The first round of survey data collection (the baseline round) occurred prior to the beginning of the 
Fall 2009 SEMAA session, at the time that individuals applied to the program. The baseline surveys 
were distributed as part of the SEMAA application packet, and were returned by families via U.S. 
mail. 

The second round (the follow-up round) occurred at the conclusion of the eight-week Fall session at 
each site, when families in both the treatment and control groups were mailed packets containing 
the student and parent follow-up surveys. Postcards and phone calls were used to prompt 
nonrespondents to complete their surveys. Nonrespondents were also given the option to complete 
the survey via an electronic form of the survey that was created to encourage participation. 

Sample for the impact study 

The impact study focused on the SEMAA sites using the Saturday model. Of the 14 active SEMAA 
sites, 7 were implementing the Saturday model at the time of this study. In a Saturday model (versus 
an in-school or after-school model), students typically attend a three-hour morning session each 
Saturday for eight consecutive weekends in classrooms located at a partnering university or other 
community locations. Parents also have the option of participating in SEMAA activities at these 
times. The Saturday model was chosen for two primary reasons. First, the SEMAA sites offering 
Saturday sessions are implementing the SEMAA program in the manner that NASA intended and 
prefers. Other models of implementation (after-school or in-school models) were adopted for local 
reasons, which are further described in Chapter Three. The after-school model differs from the 
Saturday model, in part because of the timing of the Family Cafe, and is not representative of 
SEMAA as a whole. Sites using in-school models of implementation have expanded or altered the 
SEMAA content materials for adoption as units in a school's existing science courses. Second, at sites 
using the after-school or in-school models, random assignment of students to treatment or control 
would not have been possible. Because SEMAA in-school models work by integrating the SEMAA 
curricular enhancements into a school's existing science curriculum, random assignment would have 
resulted in some student being denied access to formal science courses or units that formed a part 
of required instruction. 

The universe of students in the impact study was the pool of 4 th to 8 th grade student applicants (and 
their parents) to the Fall 2009 SEMAA session at each site participating in the impact module of the 
study. Typically, SEMAA sites are oversubscribed for the Fall session: more students apply than can 
be accommodated in the Fall eight-week session. Assignment to the Fall session (rather than the 
Winter or Spring session) is normally determined on a "first-come, first-serve" basis: those families 
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whose applications are received first are more likely to receive their first choice of sessions for the 
academic year. In many sites, prior participants receive information about upcoming enrollment 
directly from the site; new participants hear about SEMAA via advertisements, brochures, and 
SEMAA staff presentations at community or school events. Assignment in Fall 2009 was conducted 
differently because of the impact study. All applications received by a site's deadline were subject to 
random assignment to either the treatment or control group. 

The unit of assignment was the household. Each household (i.e., a student-parent pair) was assigned 
either to a treatment condition (enrollment in the Fall 2009 SEMAA session at their site) or a control 
condition (embargoed enrollment until the Winter or Spring 2010 SEMAA session). Both prior 
SEMAA participants and new SEMAA applicants were subject to random assignment. Those assigned 
to the control group were free to enroll in other (i.e., non-SEMAA) science and/or mathematics 
enrichment activities. 

Because some households included more than one eligible student applicant to the Fall 2009 
session, student applicants from the same household were assigned to the same experimental 
condition as follows: 

• Prior to random assignment, one sibling was selected at random to be the "target" student 
for purposes of the study; 

• The target student was randomly assigned to treatment or control status; 

• Any siblings of the target child who also applied to SEMAA were assigned to this same 
condition. 

Although the SEMAA project targets students in all elementary and secondary school grades, the 
study included students (and their parents) in the 4 th through 8 th grades. The 4 th and 5 th grades were 
selected to represent students at the elementary school level from whom data could reasonably be 
collected via paper and pencil survey, without requiring a one-on-one interview or a survey 
proctor. 12 Students in the 6 th through 8 th grades were targeted because they are an age group of 
particular interest to STEM educational researchers, since the science and mathematics courses 
taken in school during these middle school years may determine which types of courses are 
available for students when they matriculate to high school, and algebra in middle school is a strong 
predictor of subsequent STEM enrollment. 

The evaluation recruited 662 families who were interested in participating in SEMAA across six sites. 
Of these, 397 completed both the baseline survey that was administered prior to the Fall SEMAA 
sessions and the follow-up survey that was administered after the Fall SEMAA session. Because the 


12 Students in kindergarten and 1 st through 3 rd grades were not included in the study because they would 
typically need support completing a written survey and it would have been difficult to collect reliable 
information on the outcome measures as proposed. 
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response rate of 60 percent was lower than expected, 13 we conducted several tests to investigate 
potential nonresponse bias and found that there was no evidence to suggest nonresponse bias 
among the outcomes of interest (see Appendix A). 

Analytic approach for the impact study 

Analyses were conducted to estimate the average impact of the SEMAA program on student and 
parent outcomes. Because the SEMAA program theory postulates that the effect of SEMAA is 
cumulative across years, and the SEMAA program is designed to offer new experiences to students 
who return year after year, we also investigated whether the impact of the program on prior SEMAA 
participants and new SEMAA participants differed. 

We tested the impact of SEMAA on students separately from its impact on parents. Impacts were 
tested by comparing outcomes for the treatment group and control groups, by fitting regression 
models that accounted for the clustering of respondents within sites (multi-level modeling) and key 
covariates, including the baseline measures of each outcome and background characteristics. 14 We 
describe these models in more detail in Appendix A. 

We also conducted exploratory analyses to examine hypotheses about whether the number of 
Saturday classes or Family Cafe sessions attended was related to the outcomes of interest. Although 
the study was not designed to provide a rigorous test of these relationships, the results of these 
analyses provide suggestive evidence. 


13 The target 75 percent response rate was set with a proposed incentive for participation; however, the 
Office of Management and Budget did not approve the use of incentives for this study, and the actual 
response rate for follow-up surveys in the study was 60 percent. 

14 In models investigating impacts on student outcomes, the background characteristics included as 
covariates included prior participation in SEMAA, gender, race/ethnicity, grade, whether any family 
member was employed in a STEM field, parental employment at NASA, parents' highest education level, 
family income level, amount of help received from adults on the baseline survey, and whether English was 
the only language used in school; in addition, the baseline value on the outcome being investigated was 
included. In the models that investigated parent outcomes, covariates included student's prior 
participation in SEMAA, parent's highest education, income level, whether the parent was working for 
NASA, and whether the parent was working in a STEM field; in addition, the baseline value on the 
outcome being investigated was included. 
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Chapter 3: Implementation of the SEMAA Program 


The implementation study provides insights into how the sites implement the SEMAA components 
and select a model for implementation in response to the characteristics of their host and partner 
institutions, particular needs of their student population, and available resources. This chapter 
describes the characteristics of the sites and provides the findings from the implementation study- 
focusing on project goals, governance and structure, participant characteristics, SEMAA providers, 
models of implementation, evaluation and monitoring, and factors that influence implementation. 

SEMAA Elements 

Project goals 

The goals of individual sites are aligned with the goals of the SEMAA project and the individual host 
institutions. These shared goals were important in the establishment of individual sites, as were 
individual past experiences, and Congressional support in some cases. 

Project site goals align with those of SEMAA. The main goal of the SEMAA project, as envisioned by 
NASA, is to increase the STEM participation and retention of K-12 students traditionally 
underrepresented in these fields. This aligns well with the goals reported by the directors of project 
sites, who see their main goal as working with underrepresented children to inspire, engage and 
educate them to remain in STEM areas. As one put it, they hope to "inspire a new generation of 
explorers that are interested in STEM careers." A number of interviewees added that they are 
working with kids who would otherwise not have the opportunity to participate in this kind of 
program or be exposed to the science content and hands-on activities. 

Project site goals align with those of their host institutions. Project leads also noted that their 
project's goals have remained the same over time and align well with those of the host site. Projects 
based at postsecondary institutions tended to report that their SEMAA project goal aligns "closely" 
with that of the host institution because the university/college values outreach and diversity in 
higher education, and/or because SEMAA strengthens the school's visibility and aids recruitment. In 
contrast, projects based at public school districts/systems reported that their goals align well with 
those of SEMAA because each seeks to advance student preparation, achievement, and success. 

Shared goals were buttressed by past experience and Congressional support. Interviewees 
reported that the establishment of the individual sites was motivated by common goals as well as 
other factors. Specifically, some project directors mentioned that prior experience with related 
initiatives supported the establishment of SEMAA at their site. For example, one site director 
reported that SEMAA "built on some institutional precedents" because his college had a "successful 
history of science education and science outreach" and in fact ran one of the first programs for 
minority undergraduates in physics. Another site director noted that before establishing a SEMAA 
project his organization had partnered with NASA on a number of projects including one similar to 
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SEMAA. In contrast, other directors stated that they were motivated to establish a program because 
nothing similar had existed for their target population. These sites had established a SEMAA project 
precisely because no similar projects had ever existed at the school or in the area. Some spoke 
about the dire need for the program given the poor math and science performance of local public 
school students or the bleak economic conditions in the region. One interviewee reported that the 
area unemployment rate is 20 percent and described, "we have a large workforce that were in 
factory-based jobs and many of these factories are gone leaving the uneducated population 
unemployed. The hopes were that we could further STEM-related lessons, activities, and projects 
that would open our students' eyes so that they would come back when they were done with 
college to revitalize the area." Finally, four site directors specifically mentioned the efforts of their 
Congressional representative in helping to bring their SEMAA project to fruition. 

Project management and governance 

Most SEMAA sites are situated within the larger institutional structure of their sites, but there was 
variation across projects in how they were structured, specifically with respect to the composition of 
their central staff and their use of advisory boards. 

SEMAA projects are managed by a small group of staff. SEMAA projects are managed, on average, 
by four to five staff members. Typically, the management team consists of a site director, AEL 
coordinator, Family Cafe coordinator, and an administrative assistant; the management team does 
not include the set of instructors who teach the SEMAA sessions. Some projects also have additional 
support staff such as a Family Cafe facilitator, office manager, and perhaps another administrative 
assistant. In two sites, however, the director is the sole staff member who shoulders the 
responsibilities of coordinating the AEL and Family Cafe. 

Most SEMAA site directors hold other positions within their organizations. Project directors hold a 
variety of responsibilities, including: overseeing project implementation; managing staff; 
coordinating instruction, evaluation, and recruitment; maintaining an inventory of supplies; 
fundraising; and reporting. For about a third of the interviewees (29 percent), their work as the 
SEMAA site director or project manager is their sole position. 15 Most of the SEMAA site directors (71 
percent), however, hold other positions such as faculty member, director of a science center, or 
university staff member (e.g., department coordinator, outreach director). For these directors, as 
well as most of the other staff, SEMAA project positions tend to be part-time rather than full-time. 

SEMAA projects report to a senior institutional administrator. SEMAA projects are situated in 
various positions within the administrative structure of their grantee institutions. Within 
postsecondary institutions, projects are found in academic units such as a college (e.g., liberal arts 
and sciences college); a particular school (e.g., school of engineering, school of education) or 
department (e.g., math and science, earth and physical sciences) within a college; or an 
administrative unit or special program (e.g., office of research and sponsored programs, office of 


Two directors at public school district sites, and two at institutions of higher education. 
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precollege programs, office of academic and student affairs). As such, the site directors typically 
report to a higher-level administrator such as a vice president or dean. In the case where the 
grantee is a public school district, three projects are situated in the central office and one at a 
district science learning center; in ail cases the site director reports to senior personnel such as an 
executive director, county manager, or school board. 

Half of the active SEMAA projects have an advisory board, as did most closed sites. Of the 14 

active SEMAA sites, 7 have an advisory board, and an additional site had an advisory board at the 
beginning of their project, but it no longer meets; the closed sites interviewed had boards in place 
while they were active. Boards are typically comprised of public (school officials, teachers, 
state/district officials) and private (business, industry, parents) representatives. The frequency of 
advisory board meetings, as well as the timing of the meetings, varies across sites. Several projects 
reported that their advisory boards convened more frequently in earlier years, when the projects 
were being established and needed guidance. In later years, meetings became less frequent; at least 
one project, instead of holding regular meetings, calls upon individual board members for advice as 
needed. Other projects, however, have continued to hold regular meetings, between one and three 
times a year. One site reported holding advisory board meetings as frequently as two or three times 
per session. 

Projects where the advisory board helps with fundraising or resource procurement tended to rate 
their boards as "extremely helpful" or "very helpful" (five out of seven). Of the remaining two sites 
that had advisory boards, one mentioned that the board had provided guidance on project 
implementation and serves as a resource to the project (as guest speakers, for example), and one 
did not see its board as particularly helpful. One site director noted that their advisory board was 
particularly helpful in terms of connecting the project with experts in the field of curriculum 
development and increasing project visibility. This director explained, "what was important is that 
these are people out in the community, part of Rotary Club, Lion's Club, etc. and they helped get the 
word out about SEMAA. And now probably every engineering professor and educator knows 
SEMAA. And I have parents calling me, 'I heard from so and so about SEMAA, can I get my child in?' 

It really helped to get the word out faster." SEMAA advisory boards assist with implementation of 
activities and fundraising, but are not involved with setting policy, a common role of boards that 
might not have been necessary given that SEMAA is a well-defined program. 

Project components and implementation 

The three SEMAA project components are widely implemented across the sites, in one of three 
academic-year models— two out-of-school models (Saturday and after-school) and an in-school 
model. Sites also offer summer sessions, which are sometimes held in a different location than the 
academic year sessions. 

The three project components are widely implemented. All projects offer the three core 
components of SEMAA: (1) NASA SEMAA hands-on curriculum; (2) Aerospace Education Laboratory 
(AEL); and (3) Family Cafe. Please note that the impact study (see Chapters Four and Five) did not 
study the relative contribution of each component to measured outcomes, but rather the combined 
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effect of participation in the SEMAA program, which offers these components. The implementation 
of each component is discussed below. 

SEMAA curriculum. The SEMAA curriculum was developed to align with national standards. A 
number of sites reviewed the lessons to confirm this and to verify whether the curriculum aligned 
with state standards as well. 16 Their site directors concluded that the SEMAA curriculum aligns well 
with national, state, and even local standards. Many projects complement the SEMAA curriculum 
with additional experiences designed to meet local needs or interests. For example, at least half of 
the sites specifically cited the use of field trips (64 percent) and guest speakers (50 percent). A 
smaller number mentioned incorporating robotics, in some cases through a club or competition (21 
percent), mentorship (14 percent), and career exploration activities (14 percent). A few projects 
offer enrichment activities through collaborations with other programs like Young Eagles, Women in 
Aviation International, and university engineering clubs. 

Aerospace Education Laboratory. AELs, which exist at every site, are predominantly located on 
university and college campuses (64 percent), in such places as the library, science center, and 
engineering school. Other locations for the AEL include a large classroom at a middle school, the 
hosting science center/museum, and a school district facility. One project has two AEL sites, one 
located at a public high school and another at a university. Half of the sites reported offering all or 
nearly all 10 SEMAA simulations (wind tunnel, flight planning, flight simulator, GPS, remote sensing, 
aeronautic interactive, aircraft design, etc.) as part of their core curriculum. One site explained that 
not all grades they serve conduct every simulation because the site tailors offerings by age group. 

Family Cafe. A Family Cafe is offered by all sites, typically at the same location where classes are 
offered. 17 Most of them offer the Family Cafe on a weekly basis (77 percent), while the remainder 
offer less frequent sessions (e.g., once a month or once a quarter). Most sites run the Family Cafe 
concurrently with the student academic sessions (69 percent) to encourage participation. Estimates 
of the proportion of families that attend Family Cafe activities range from 15 to 85 percent, with a 
mean of 48 and a median of 40 percent. Typical Family Cafe activities include guest speakers, 
workshops, hands-on activities, and field trips. Examples of topics covered include aviation and 
astronomy, health care, financial management, parenting skills, high school course selection, college 
applications, time management, testing standards, how to use the Internet, and locally available 
resources and programs for students. One site director observed, “The parents love anything related 
to technology. They also love anything related to helping their kids move further in school." Another 
described these meetings as "family festivals" because a student's entire family may attend— 
parents, grandparents, aunts, uncles, and siblings. Several sites deliberately work in some time for 
parents and students to do hands-on activities together. At one site during the last 20 minutes of 


16 One site stated they hired a curriculum specialist to study alignment in an effort to achieve buy-in from 
local public school teachers. 

17 One exception is a site that offers the Family Cafe in a building across the street from where the student 
sessions are held. 
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Family Cafe parents go to their child's class where the teacher will provide an overview of that day's 
activities. 


Three models of implementation prevail. SEMAA was designed as a Saturday program (Saturday 
model), but in response to local needs, some projects adapted sessions to be offered during the 
school day (in-school model) or after school hours (after-school model). The same site or location 
may engage in more than one implementation model at different times within a year or different 
years. The use of these models across sites is discussed in greater detail in the next section. 
Currently, seven sites characterize themselves as being primarily a Saturday program (50 percent), 
six provide in-school services (43 percent), and one runs as an after-school program. 

Projects typically offer three academic sessions during the school year and a summer session. The 
academic year sessions vary greatly in format, especially in the case of the in-school model. For 
example, of the sites that use an in-school model, one site holds 55 classes of 120 minutes during 
the course of a single academic session while another holds 4 classes of 480 minutes. Less variation 
is found across the sites that use the Saturday model; these projects each hold 7 or 8 classes of 180 
minutes. The one project that currently implements an after-school model holds 8 sessions of 150 
minutes. During the summer, half of the 14 active SEMAA sites offer four one-week sessions, with 
the remaining sites offering two to three one-week sessions (3 sites), one five- to six-week session (2 
sites), and one or two two-week sessions (4 sites). 18 

Academic sessions and summer sessions may be held at different locations. SEMAA projects 
generally hold their academic-year classes in public schools (50 percent), on the campuses of 
postsecondary institutions (36 percent), both (7 percent, or 1 site), or at a science center/museum 
(7 percent). In some cases the venue for classes changes during the summer months. For example, 
one project usually delivers SEMAA classes in public school classrooms either during or after school 
but in the summer SEMAA classes are held at the university. Similarly, at another site SEMAA is held 
at a science center during the school year, but accommodates more students during the summer by 
holding classes at a middle school. 

Sites undertake multiple outreach efforts. SEMAA projects conduct outreach through a variety of 
efforts. All but one of the sites are engaged in outreach through the AEL. Typically a site will target 
K-12 students for outreach by hosting school field trips and campus tours at the AEL. Other sites 
mentioned specifically serving teachers, parents, or the entire community. For example, one urban 
site reported that teachers from surrounding school districts use the AEL after the school day 
(approximately 480 non-SEMAA teachers participate annually). Estimates of the number of 
individuals reached annually through such activities ranged from 0 (rural site) to 5,500 (urban site), 
with the median being 775 students. In addition, project staff make classroom visits— bringing 
portable star lab, e-planetarium, AEL laptops, space suit, etc.— and have made appearances at area 


Counts add up to two more than the universe of current sites because two sites appear in two of the 
categories listed. 
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science fairs, career fairs, summer camps, Boys and Girls Club, hospitals, church daycare, and 
homeless shelters. SEMAA projects also work with science clubs and robotics teams. 

Students served 

On average, SEMAA sites serve 1,495 students a year (median of 1,169), ranging from 431 to 3,100 
depending on the site. 19 Collectively, during the 2007-2008 school year, the 14 SEMAA sites served 
20,929 direct student participants from 205 school districts and enrolled at 551 elementary schools, 
254 middle schools, and 141 high schools nationwide. These estimates, reported by site directors for 
this study, are close to those reported in sites' quarterly reports, which document that almost 
19,000 students are served from 158 school districts and 576 elementary, 233 middle and 150 high 
schools. The only notable discrepancy between the two different sources is in the number of school 
districts reported. 

Presentations and word of mouth are seen as the most effective recruitment strategies. Projects 
recruit students through a number of means, most commonly through newspaper advertisements, 
flyers, presentations at schools and other community venues, and word of mouth (parents and 
teachers). In-person presentations and word of mouth were the most frequently cited methods of 
effective recruitment. Several sites noted that their project is so well known and in demand now 
that they no longer need to recruit. Although projects usually use some sort of application or 
registration form, there are no selection criteria and projects accept students on a first come, first 
served basis. Projects only turn away students when they have reached capacity. Projects have 
devised strategies to address capacity limitations. One project, facing overwhelming demand, has 
limited student participation to one academic session and the summer session. Another one limits 
summer session participation to those who did not participate in the regular school year sessions. 
These approaches have limitations, however, to which projects are responding as well. Due to 
budget cuts, a number of sites reported scaling back their summer offerings, and partnering with 
various community groups to offer sessions at existing camps or programs. 

Most projects serve students in all grades, but those located at colleges and universities are 
somewhat less likely than public school districts to serve early elementary grades. Most projects 
(10 out of 14, or 71 percent) serve students from kindergarten through 12 th grade. The other sites, 
however, serve a subset of these grades. Since all sites serve grades 2 to 8, variation was driven by 
grades at the two extremes, at the beginning of elementary school (grades K and 1) and in high 
school (grades 9 to 12) (see Exhibit 5). Kindergarten is the grade level least likely to be served, 
followed by grades 11 and 12. Public school districts are somewhat more likely to serve kindergarten 
and 1 st grade than colleges and universities. 


19 These figures refer to "direct student participants" and are based on a 12-month period— including all 
sessions offered during an academic year plus the summer sessions. Averages were calculated based on 
the estimates provided by each site included in this study. 
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Source: Telephone interviews with 14 open sites. 


SEMAA projects report serving minority (mostly African Americans, but also Hispanics and Native 
Americans) and economically disadvantaged youths. Twelve of the 14 active sites serve mostly 
African American students. In these sites, African Americans comprised 58 to 94 percent of students, 
or an average of 78 percent. Among the two remaining active sites, one serves Native Americans 
exclusively (100 percent) and the other primarily Hispanics (85 percent). The directors reported that 
the students served tend to come from disadvantaged backgrounds. On average, about 60 percent 
of them come from families living below the poverty line, ranging between 15 and 96 percent 
depending on the site. These characteristics align with the goals of the SEMAA project. 

SEMAA projects encourage student participation overtime. Based on site estimates, in the out-of- 
school models a mean of 65 and median of 75 percent of participants continue participating in 
SEMAA from one year to the next, ranging between 40 and 90 percent per site. This estimate 
excludes sites implementing the in-school model, because participation in the in-school model is 
mandatory for all enrolled students, and attrition is driven by school mobility. 20 On average, about 
half 21 of SEMAA academic-year participants will enroll in a summer session (ranging from 0 to 90 
percent). This pattern of reenrollment is consistent with SEMAA's program theory of taking a 
longitudinal approach to engaging students and offering STEM opportunities across multiple years. 

Projects report higher attrition in the upper grades. Nearly half of the sites pointed out that 
attrition is greater at the upper grade levels than in earlier grades, as students in later grades are 
less likely to re-enroll in the program from one year to the next. One site director observed, "The 
greatest attrition is at the high school level. They don't want to get up on Saturday mornings, they 


Two sites that follow the in-school model provided estimates of continuity among students in their 
Saturday or after-school programs. Including those sites in the calculations yields the following estimated 
percentages: 63 mean; 65 median; 35 minimum; 90 maximum. 

Mean of 52 and median of 50 percent. 
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have a job, do sports, or are just not interested anymore." Another explained that "we get lower 
enrollment in the upper grades and that is why we target high schoolers more than lower grades. 
After the middle school it starts slowing down." Aside from seeking to put together sessions that are 
interesting, fun, and stimulating, some things that projects are doing to ensure continued student 
participation is calling parents to impress upon them the importance of participation, identifying 
student role models and highlighting their success (internships, scholarship, college acceptance), 
and offering a guaranteed spot for next year to those who achieve good attendance. 

SEMAA providers 

SEMAA provides training to instructors, often certified teachers, to implement the SEMAA 
components. Trained SEMAA instructors generally continue with the SEMAA program from year to 
year. 

Sites have no difficulty recruiting teachers to lead classes. Projects reported they did not encounter 
difficulties in recruiting enough qualified teachers to lead the SEMAA classes. The projects with an 
in-school model benefit from district assistance in hiring teachers, and those with an out-of-school 
program rely on referrals from current teachers. In general, in-school site directors work with a 
school principal or liaison within the school system to select teachers who are hired by the district. 
Most projects reported hiring only inservice teachers (64 percent), while the others rely mostly on 
inservice teachers but also employ some preservice teachers. One site reported also using college 
professors and graduate engineering students to teach sessions. In hiring SEMAA teachers, site 
directors generally select teachers based on criteria such as whether they have a math or science 
background, state certification, advanced degree or extensive teaching experience, and enthusiasm 
for the subject matter and working with children. 

Projects report low teacher turnover. Site directors indicated that teacher attrition from one year 
to the next is low, as instructors tend to stay with the SEMAA project for a long time. Many directors 
stated that they have largely the same group of teachers that they had at the start of the project. As 
one put it, "The same teachers participate year after year." If instructors leave, it is usually because 
they move, get promoted, or face personal constraints (e.g., family, time). As the site director for a 
Saturday project stated, "The same first group of teachers is still here. That is one of the things that 
makes our program successful." 

Teachers across sites are trained to deliver the SEMAA curriculum. All sites engage existing 
teachers to train new teachers, but the training varies in terms of size (group versus individual), 
structure (highly structured versus informal review and questions/answers) and duration (one-day 
session, several days of shorter sessions, one-year assistantship). For the majority of sites training 
sessions are carried out by grade level and are held at the start of each quarterly session, but vary 
greatly in duration. For example, one site conducts training in one full day, while another conducts it 
in multiple meetings spread across three weeks. In some cases, training entails going through each 
lesson of the curriculum, covering both content and pedagogy. In other instances it is less detail 
oriented, and is structured as an opportunity to review the materials and to ask questions. A handful 
of projects specifically mentioned adopting a "train the trainer model" (36 percent) in which the 
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original set of instructors trained by the national SEMAA office now train the new instructors that 
come on board. For a few sites this may be a one-on-one process where the new teacher observes 
how a class operates and then meets with the teacher they are replacing for individual training. A 
couple of sites complement basic training with a teaching assistantship, ensuring that new 
instructors assist a SEMAA teacher in the classroom for a year before being assigned their own class 
to teach in the second year. 

External support, partnerships and collaborations 

Sites were asked about the collaborations and partnerships that contributed, either financially or in 
some material way, to the implementation of their projects. Naturally, site directors spoke about 
the internal partnerships that they have formed at the host institution that provide in-kind 
contributions (e.g., physical space, material resources, administrative support). Of particular interest 
are the external partnerships and collaborations that have been fostered beyond the financial and 
in-kind contributions provided by NASA and the host institution. On average, sites raised significant 
sums to support the SEMAA project. These funds were needed, in conjunction with those provided 
by NASA, to sustain the projects. None of the sites felt, however, that their fundraising was yet 
sufficiently successful to allow them to become independent of NASA. Nonetheless, we report here 
noteworthy information regarding fundraising sources and differences across sites. 

SEMAA projects received support from multiple external sources— community groups, industry 
and research centers. External sources of support for SEMAA projects come from school districts, 
colleges and universities, research and science centers, laboratories, industry, and community 
groups. Exhibit 6 shows the proportion of sites that developed such external collaborations. When 
the host site is a university or college, a public school system becomes an "external" partner, and 
vice versa when the host is a public school district. Excluding partnerships that constitute the 
foundation of the SEMAA project, 22 and focusing instead on "other" partners and collaborators, 
reveals that sites at public school districts are more likely to develop collaborations with community 
groups and museums/centers, whereas college and university sites are more likely to develop ties to 
other institutions of higher education and industry. 23 Specifically: all four public school districts that 
run SEMAA projects developed collaborations with community groups; only three of the nine 
colleges/universities partnered with community groups; most (five of eight) of the 
college/university 24 sites developed collaborations with industry, as did two of the four public school 
districts; and two public school sites and two university sites developed collaborations with a 


These partners are those that provide a resource or service that is an integral component of SEMAA. 

23 Measured by the number of partnerships/collaborations established. Note that this general conclusion is 
supported by data gathered from the telephone interviews and the quarterly reports prepared by 
Paragon TEC, Inc. The more detailed information on specific collaborations and partnerships is drawn 
from interviews, which yielded somewhat different distributions compared to the information contained 
in the quarterly reports. 

24 Excluding community colleges. 
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museum or science research center. Only one site had not developed any external partnerships or 
collaborations, at least in the 2007-2008 school year. 


Exhibit 6. Partnerships at Active SEMAA Sites 
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Source: Telephone interviews. 


Institutions of higher education raise more funding through collaborations and partnerships than 
public school districts, on average. This is true of external and internal funding and is supported by 
both available data sources, namely, evaluation telephone interviews and quarterly reports. The 
point estimates obtained from these two sources vary, but both suggest that institutions of higher 
education raise greater amounts of funding than public school systems. For example, all four public 
school districts that run SEMAA projects leverage funding from community groups, ranging in value 
from $360 to about $106,000, compared to the $111,000 to $150,000 leveraged by colleges and 
universities. Similarly, partnerships with industry are estimated to bring contributions of $5,000 to 
the public school districts and between $15,000 and $600,000 to the universities. 2 " In the case of 
internal support, sites at school districts leverage about half of the funds obtained by sites at 
institutions of higher education. 26 

Parents are a resource at a few sites. Although not a formal partnership or collaboration, parents 
occasionally assist in the classroom as well as in AEL and outreach activities. At the site where there 
appears to be the most parent involvement, parents select and organize field trips, coordinate 
family night (where families engage in hands-on STEM activities together), and take part in home- 
based family activities (e.g., watching a video about science and NASA). 


25 These estimates are based on data gathered through telephone interviews and differ from those obtained 
from the quarterly reports submitted to NASA. This is not surprising given that respondents must rely on 
memory and often round when reporting verbally. But the general conclusions reported here hold 
regardless of data source. 

26 Estimates range from about 40 percent (telephone interview data) to 67 percent (quarterly reports). 
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SEMAA Implementation 


The use of the three academic implementation models introduced above— Saturday, after-school, 
and in-school— varies across sites, both in which model they are currently using and in how the 
models have been used overtime. 

SEMAA models and model selection 

SEMAA sites can be classified into three models defined by the timing of when they offer the SEMAA 
components— Saturday, after-school, or in-school. In addition sites can be characterized by the 
constancy from year to year with which they have implemented a chosen model. Various contextual 
factors drive sites' decisions to select a model, and there is some evidence that the in-school model 
is used in response to student demand and sustainability. 

Models of implementation are defined by when sites offer the SEMAA components. A research 
question central to this study was whether distinct models of implementation could be identified. 
Often in national programs, variation in implementation arises as program components are offered 
at some sites but not at others. However, this was not the case with SEMAA, as all sites offer the 
three SEMAA components. Instead, the variation— the different models— was grounded in the 
timing of when sites offer the three SEMAA components as prescribed by the SEMAA project. In 
other words, the main models are characterized by the type of session— Saturday, in-school or after- 
school— which is partly related to resource availability and contextual factors (described below). The 
Saturday and after-school models are both out-of-school programs, which share key characteristics: 
they are offered outside of regular school hours, have as their central goal the delivery of the 
SEMAA content, are "special" programs (as opposed to part of a regular science class), require the 
hiring of staff, and require special transportation (children must be picked up from school late for 
after-school programs, or driven on Saturday to the project's location). These models differ from the 
in-school model in that they do not have a captive audience, and instead are attended by individuals 
who choose to participate in the program. Conversely, the in-school model includes both those 
individuals who would have chosen to engage in SEMAA outside of school hours as well as 
individuals who participate because it is part of the science curriculum at school. 

A longitudinal perspective reveals within-site changes in models implemented. The 

implementation of these models over time reveals another dimension important to the 
implementation of SEMAA— a dimension introduced by the passage of time and the desire to 
institutionalize the projects (i.e., continue beyond NASA funding). The models selected and 
implemented within sites over time suggest that project sites can be classified according to the 
constancy of the model they implement into one of three types— constant, transitioned, and in 
transition. This typology is based on whether sites have changed or are exploring changing their 
model type. A "constant" site is one that has not changed its model type since its inception (e.g., has 
only offered Saturday sessions); a "transitioned" site is one that originated in one type but has 
shifted to a new model (e.g., began implementing a Saturday model and is now implementing an in- 
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school model); and a site "in transition" is one whose offerings indicate that they are exploring other 
models, although still offering the original session type. 

According to site directors' characterizations, among the active sites six have remained constant, 
four have transitioned to a different model type, and four are in transition status. Six sites are 
classified as constant because they are still implementing the model they were implementing at 
their origin; among these, three are implementing the Saturday model, two the in-school model, 
and one the after-school model. 27 Four sites belong in the in-transition category, because they are 
exploring alternate models; each of these started with the Saturday model. Three are exploring the 
in-school model to help with sustainability or respond to demand. Two are exploring the after- 
school model. Finally, four sites are classified as transitioned. Each of these has moved to the in- 
school model for issues of sustainability; in one case the transition was in the initial plans of the site. 
Three of these sites began with the Saturday model, and the remaining site began with the after- 
school model. 

This longitudinal look at model implementation revealed that sites tend to implement the program 
as envisioned by the SEMAA project (as a Saturday model)— but in response to (a) student demand 
for participation or (b) funding pressure (as they approach the end of their funding or need to 
expand in response to student demand)— projects either transition to an in-school model or close 
(or both). A variant of these alternatives is found at the only site that successfully graduated from 
NASA funding and that, to this day, continues offering a limited set of activities through outreach. 
This site (a) originated as a Saturday model, (b) evolved into a pseudo in-school program (as 
teachers continued to seek activities offered by the center and to use the SEMAA materials and 
applications after the site was no longer funded by NASA SEMAA), and (c) later focused on outreach 
activities that they expect to be able to continue providing because these align with their mission. In 
other words, by conducting AEL outreach, this center has been able to sustain some of the SEMAA 
activities. 

The inclusion of closed sites in this longitudinal analysis reinforces this observation. Exhibit 7 shows 
the transition of the universe of SEMAA sites (25 closed and open projects) over time. Organized by 
model type, the figure shows the model under which each site was established (18 Saturday, 2 in- 
school and 5 after-school), and the predominant model they describe offering at present (if open) or 
offered at the time they closed. 28 Of the 18 sites that began as Saturday programs, only 7 continue 
to offer Saturday sessions, 6 transitioned to in-school programs, 29 and 5 closed. Similarly, of the 5 
sites that opened as after-school programs, 1 remains, 1 transitioned to in-school, and 3 closed. This 


One site offering the in-school model was at one point in transition (exploring after-school offerings) but 
due to resource constraints discontinued that model and continued with the original in-school offerings. 

28 Please note that sites may offer more than one model. This analysis focuses on the model that site 
directors consider their predominant or most commonly offered model. These classifications differed only 
in one instance from classifications made by the National SEMAA Office, which would classify one of these 
in-school sites as predominantly offering a Saturday model. 

29 Three of them later closed, and one continued with more limited offerings (AEL outreach). 
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suggests that the temporal transition that the national SEMAA office envisioned from a (preferably) 
Saturday program financed by NASA to a self-sustaining one, has evolved into a transition to closure 
or in-school services. 30 


Exhibit 7: Primary SEMAA Models at Inception and Present 
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Christian Fenger Academy 
CUNY Medgar Evers College 
CUNY/ York College 
Cuyahoga Community College 
Fernbank Science Center 
Lakeland Community College 
Morgan State University 
University of the District of Columbia 
Warren County Fligh School 
Wayne State University 
Wilberforce University 
Wiley College 
Albany State University 
Richland County District One 
Sinclair Community College 
Tennessee State University 
University of MD, Eastern Shore 
Winston-Salem State University 
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Note: Five of eleven closed sites were not interviewed. 

Source: Telephone Interviews (2009), Quarterly Reports, Standard Operating Plans (SOPs). 


30 Of the 11 closed sites, 3 closed due to funding issues, and 4 closed due to administrative challenges or 
inadequate student demand for the Saturday program; no explanation was obtained for the closings at 
the remaining sites. 
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Facilitating implementation 


Committed staff and institutional support were the most commonly cited factors facilitating general 
implementation, whereas funding was the key barrier mentioned. One site director stated, "We 
have an outstanding staff that is committed to the children and the parents. They are not making a 
lot of money here. They are truly inspired to keep the program going." Another site director who 
reported having a "highly qualified" and "committed" staff noted that "screening your staff is 
critical." Similarly, a closed site mentioned specifically that lack of committed staff was a critical 
barrier to implementation. Site directors also mentioned that having the support of the host 
institution (school district or university) is critical, a comment echoed by a closed site that lacked 
such support. But in terms of the biggest challenges to project implementation, site directors 
overwhelmingly cited funding difficulties— whether directly (to maintain the project) or indirectly (to 
staff it adequately given demand). They explained that funding cuts and lapses adversely affected 
their projects, particularly given the lack of prospective financial partners to draw upon in their local 
areas, and cited difficulties in fundraising (79 percent of the 14 current sites). Several spoke about 
how inadequate staffing, stemming from funding constraints, impairs their ability to expand their 
project to serve more students. 

Sites also discussed the factors that influence their selection of model types, and explained the 
transition to new models over time within sites. Most sites begin as Saturday models under NASA 
sponsorship. Sites have mentioned, however, two types of constraints that have led to modifications 
or adaptations of program offerings— namely, transportation and resources. The first limitation- 
transportation— is faced by rural sites, where students may be spread across great distances and 
parents may have difficulties coming to the sites to participate in the Family Cafe during the week, 
or picking up children participating in after-school programs. At these locations, Saturday offerings 
provide the greatest probabilities of both student and parent participation. Similarly, site directors 
mentioned that the additional costs associated with out-of-school programs (in particular teacher 
salaries) make the provision of those programs without NASA funding difficult. Hence the high 
incidence of site closings and transitions to in-school models among sites. 31 

Monitoring and evaluation 

Projects tend to restrict evaluation activities to those required by NASA for reporting purposes . 32 

Projects gather basic data (such as participant demographics) to report to NASA and generally use 
these data for their own planning and fundraising purposes. For example, some projects mentioned 
that data guide the identification of populations to be targeted for recruitment or project growth 
(e.g., if there are demographic groups that are not well represented), to seek external funding and 
write proposals, and to improve project offerings. 


Not all sites closed due to financial problems; lack of adequate student demand and administrative 
difficulties were other reasons. 

Exception for the monitoring of student progress, described below. 
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Sites conduct monitoring of student progress only on a limited basis, with a few notable 
exceptions. Student progress is not measured directly, but at some sites it is monitored indirectly 
through regular school grades, and a couple of sites take more rigorous approaches. Twelve of the 
14 active sites mentioned that they do not monitor student progress, but 3 of them (in-school) 
added that they do rely on student school grades to follow student progress and infer the impact of 
program participation; this is also true of a Saturday program, but for the upper grades only. The 
remaining two active sites take a more rigorous approach to progress monitoring. The project 
directors at these Saturday sites stated that they conduct pre and post testing of students at the 
beginning and end of each session, respectively, to measure knowledge growth. Lastly, four sites 
mentioned other specific monitoring and evaluation efforts. One did a study of high school students 
to measure changes in interest and achievement, and found improvements after students 
participated in their eight-week SEMAA session. Another takes advantage of regular standardized 
testing to compare average achievement among participants and nonparticipants (e.g., in middle 
school earth-space). 

Projects typically do not track their participants, with a couple of exceptions among institutions of 
higher education. In general, projects have not followed their participants' progress, an exercise 
that requires funding and technical expertise that project staff might lack. One site has tracked their 
12 th grade participants into college, and another one stated that in the summer they contact parents 
to find out the field and college into which 12 th grade participants got accepted. 33 A couple of others 
made similar attempts by telephone or mail, but discontinued them because they lacked the staff to 
conduct follow-ups or did not obtain a good response rate in their mail inquiries. Not surprisingly, 
reported attempts at tracking participants were all conducted by institutions of higher education, 
which are likely better equipped to conduct such work than public school districts. 34 

Site directors perceive positive outcomes for students participating in SEMAA. Although projects 
do not conduct outcome evaluations, they cited anecdotal evidence of increased student interest in 
STEM, improved performance in math and science (over time and/or compared to their non-SEMAA 
peers), increased student exposure to the fields of and opportunities in STEM and in NASA, 
enhanced parental education regarding these fields and opportunities, and greater resources and 
training provided to teachers. 

Sustainability 

Interviews revealed that there are two related issues regarding sustainability— namely, financial and 
mission sustainability. This section discusses both. 


However, no estimate of the proportion of participants who go on to STEM in college was provided. 

34 Only two sites could estimate the approximate share of their 12 th graders who enroll in college in STEM 
(45 percent in one, and 50 percent in the other), but in one case it was based on only one cohort because 
they discontinued tracking. 
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The in-school model is perceived as a viable sustainability plan. As discussed earlier, most of the 
sites still open today began as Saturday programs, but several of them transitioned to an in-school 
program and others are beginning to offer in-school sessions. Site directors mentioned that they 
shifted or are in the process of shifting to an in-school model for cost reasons, to increase 
participation, or to prepare to sustain the program post-NASA funding. "One of the main concerns is 
that SEMAA programs have no long-term funding plan... So, if our funding is to stop, we would still 
continue the in-school model." 

Some projects combine offerings across models to leverage resources and reach more students. 

For example, a self-proclaimed successful Saturday program is offering in-school sessions to reach 
more students and because "it can be done pretty inexpensively, as teacher salaries are covered [by 
the school district]." In contrast, an in-school site discontinued its after-school offerings which 
required funding for staff salaries. 

More isolated areas encounter greater difficulties in fundraising. Projects located in more remote 
or rural areas do not have easy access to partners and potential resources, as might be the case in 
sites located in more populated urban or suburban areas. As one director put it, "We are not a 
hotbed of industry here, and we don't have all the instant partners anywhere near us." 

Site directors assume the role of educator-administrators versus fundraiser. Most site directors do 
not see fundraising as one of their main responsibilities. Indeed, when asked to summarize their 
responsibilities, only three of them mentioned fundraising. More to the point, one director pointed 
out that "I cannot run the program and fundraise and teach and do everything else." Clearly, the 
projects directors, most of whom also hold other positions/responsibilities, find tension between 
their role as administrators and academic leaders, and the expectation that they seek funding for 
the future. 

Some site directors fear that the science and NASA foci of the SEMAA program might be put in 
jeopardy by the in-school approach. One director worried that in-school programs might not 
dedicate the requisite time to the science curricula when pressured to focus on mathematics and 
reading. This director stated, "Science is being put on the shelf in [district name deleted] because 
they are looking at reading and mathematics." This concern is unsurprising given that these are the 
subjects tested for NCLB 35 purposes. Another interviewee echoed this concern stating, "you're 
looking at schools... [not] adequately meeting AYP, most schools are in crisis, trying to improve math 
and reading scores, so science is left out ." 36 


35 Refers to the No Child Left Behind Act, federal legislation enacted in 2002 that imposes strict testing 
requirements in schools throughout the nation. 

36 At this school, time dedicated to SEMAA has been mandated by the leadership in an attempt to prevent 
the crowding out of science. AYP stands for Adequate Yearly Progress, a concept introduced in NCLB to 
measure average school performance over time. 
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Chapter 4: Characteristics of SEMAA Participants 


The outcomes study was conducted to investigate the impact of the SEMAA program on participant 
outcomes. As described earlier, this study module was conducted at SEMAA sites that implement 
the original Saturday model. In this chapter, we discuss the characteristics of the students in grades 
4 through 8 who applied for the Saturday model of SEMAA at 6 of the 14 active SEMAA sites in the 
Fall of 2009. These sites comprised all but one of the eligible sites (i.e., those running a Saturday 
model) and close to half of all active SEMAA sites. Although not all models or grades are included, 
these data provide insight into the students who are attracted to and apply to the SEMAA program. 
In this chapter, we present descriptive statistics on student and parent background characteristics, 
and prior experiences with SEMAA. We end with a discussion of the study sample at baseline, 
describing the outcomes used in the outcome study and discussing the baseline levels of these 
outcomes. 

Background of Participants 

Overall, background characteristics were similar across the treatment and control groups. Appendix 
A discusses the baseline balance tests conducted on the analytic sample to ensure that the 
treatment and control groups, as well the pre-test and post-test samples, were similar. 

Background characteristics of the baseline sample are presented in Exhibit 8. Slightly more than half 
(55 percent) of the students were male. In line with SEMAA's mission of serving groups traditionally 
underrepresented in STEM, a majority of students (72 percent) identified themselves as belonging 
to a racial or ethnic group traditionally underrepresented in STEM fields: 64 percent identified 
themselves as Black or African American, 11 percent as Asian, 8 percent as Hispanic/Latino(a) and 6 
percent as White. Applicants were predominantly English-speaking— 79 percent spoke only English 
at home and an additional 19 percent spoke English and another language at home. Similarly, 87 
percent spoke only English at school, with an additional 12 percent reporting they spoke both 
English and another language at school; Spanish was the predominant other language spoken. 
Finally, just under half (49 percent) of applicants reported an annual household income of over 
$50,000. This is more than double the 2009 national poverty threshold of $22,050 for a family of 
four living in the 48 contiguous states or the District of Columbia, 37 and differs slightly from sites' 
reports of serving a majority of individuals from below the poverty line. 
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Exhibit 8: Student Characteristics 



N 

Percent 

Gender, n=662 

Male 

364 

55% 

Female 

298 

45 

Grade, n=662 

Four 

168 

25% 

Five 

173 

26 

Six 

142 

21 

Seven 

103 

16 

Eight 

76 

11 

Ethnicity, n=609 

Hispanic/Latino(a) 

49 

8% 

American Indian/Alaska Native 

2 

0 

Asian 

73 

12 

Native Flawaiian/Pacific Islander 

2 

0 

Black/African American 

421 

69 

White (Non-Hispanic) 

41 

7 

Other 

21 

3 

Any Prior Participation in SEMAA, n=662 

Yes 

389 

59% 

No 

273 

41% 

Number of Students in Family Who Applied to SEMAA in Fall 2009, n=662 

One 

439 

66% 

Two 

186 

28 

Three or more 

37 

6 

Estimated Annual Household Income, n=521 

Under $50,000 

265 

51% 

Over $50,000 

256 

49 

Language Spoken at Home n=652 

English 

515 

79% 

English and Something Else 

124 

19 

Something Else 

13 

2 

Language Spoken at School n=650 

English 

565 

87% 

English and Other Language 

77 

12 

Other Language 

8 

1 

Note: Percents were calculated from non-missing data. 


Additional data were obtained on the educational and professional backgrounds of the parents or 
guardians of SEMAA applicants. The parents who responded to the survey were notably relatively 
highly educated; 93 percent had completed at least some college or university courses. Over two- 
thirds of respondents (66 percent) had completed at least a bachelor's degree and over one-quarter 
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of respondents had at least one graduate degree (28 percent). Although almost no parents were 
direct employees of NASA, one-third of applicants had a parent who worked in a STEM field. 


Exhibit 9: Parent Characteristics 



N 

Percent 

Highest Level of Education, n=628 

Some high school 

10 

2% 

High school (diploma or GED) 

35 

6 

Some college or university 

110 

18 

Associates degree 

61 

10 

Bachelors degree 

167 

27 

Some graduate school 

69 

11 

Graduate degree 

176 

28 

Works in a STEM Field, n=503 

Yes 

166 

33% 

No 

337 

67 

Currently Works for NASA, n=617 

Yes 

4 

1% 

No 

613 

99 


Baseline Levels of Student Outcomes 

Students' values on outcomes of interest were measured at baseline, that is, at the time that 
individuals applied to the Fall session of SEMAA. Reasonable short-term outcomes were selected 
that followed from the program logic model presented earlier, to investigate students' engagement 
in science, self-efficacy in science, interest in science, interest in pathways to STEM careers, and 
support for the importance of STEM in society. Exhibit 10 presents the baseline levels on each of 
these outcomes. The indicators used to measure these outcomes are each described below. It is 
noteworthy that some of the baseline estimates were high on several of these outcomes, making it 
potentially difficult to detect effects of the program on these outcomes. 


Exhibit 10: Student Baseline Responses 


(n=662) 

Outcome 

Values 

Mean 

sd 

Desire to do science 

1-4 

3.15 

0.53 

Participation in informal science activities 

0-5 

2.35 

1.19 

Interest in informal science activities 

0-4 

3.25 

0.60 

Engagement in STEM activities 

0-1 

0.51 

0.29 

Anxiety toward science 

1-4 

1.58 

0.51 

Self-confidence in science 

1-4 

2.75 

0.29 

Interest in high school STEM coursework 

0-4 

2.67 

0.98 

Interest in college STEM coursework 

0-4 

2.84 

0.89 

Interest in STEM careers 

0-4 

2.40 

0.82 

Value of science to society 

1-4 

3.28 

0.45 
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Desire to do science 


A student's desire to do science represents his/her level of motivation to engage in science-related 
activities. This was measured using items 38 about how much students agreed with statements that 
they enjoy learning about science, as well as whether they independently engage in science-related 
learning activities, such as reading about science aside from their schoolwork. Each item was 
measured on a Likert-type scale, with four response categories ranging from "strongly disagree" to 
"strongly agree." This composite indicator had a range of possible values from 1 to 4. On average, 
students at baseline scored 3.15 on this measure, which falls between the "agree" and "strongly 
agree" categories, indicating that the average SEMAA applicant was strongly motivated to engage in 
science-related activities. 

Participation in informal science-related activities 

Students' participation in a variety of informal science-related activities was represented by a 
composite of 10 questions that each asked students to report on the frequency of their involvement 
in a given activity over a three-month period, 39 using a six-point scale of 0 to 5— "not at all," "one 
time," "about once a month," "about twice a month," "about once a week," or "more than once a 
week." The activities included going to a science museum or zoo; reading about science in a book, 
magazine, or on a website; taking something apart or building something new; watching a science 
program on television; trying to figure out how something works; doing an experiment; learning to 
use a new tool; using a science kit or doing a science project at home; and using a computer or 
technology to solve a problem. This composite indicator had a range of possible values from 0 to 5. 
Across the 10 activities, on average at baseline, students scored 2.35. On average, at baseline 
students had participated at least once in the previous three months in seven to eight of the listed 
activities. 

Interest in informal science activities 

Students' interest in engaging in informal science activities was measured by having students rate 
their interest in 10 out-of-school science activities using four response categories: "I would really 
not like it," "I would not like it," "I would like it" or "I would really like it." These 10 activities were 
the same that students rated for frequency of participation, as listed above. This composite 
indicator had a range of possible values from 0 to 4, and students scored on average 3.25 at baseline 
on this scale, indicating a fair amount of interest in these activities among applicants. 


Items for the desire to do science, anxiety toward science, self-confidence in science, and value of science 
to society composites were adapted from the Modified Attitudes towards Science Inventory (mATSI) 
(Weinburgh and Steele, 2000). 

At baseline students responded how often they had engaged in these activities over the summer, and at 
follow-up were asked how often they had engaged in the same activities over the last three months. 
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Change in interest in informal science activities 


In addition to measuring levels of interest in informal science activities, the follow-up survey asked 
students directly about changes in their interest in these activities compared to how they felt over 
the summer. This indicator was designed to measure self-perceived changes among students in the 
study, and was included to guard against the possibility of a ceiling effect. This was a potential 
concern, since it was possible that the students applying to SEMAA would already have high baseline 
levels of interest in these activities. If baseline interest in informal science activities was too close to 
the highest possible value, then a change between baseline and follow-up values would be more 
difficult to detect. This indicator provided an alternate measure of the change in interest in informal 
science activities between baseline and follow-up. Students rated the same 10 activities in this 
measure that they rated both for interest and frequency of participation. This composite indicator 
had a range of possible values from 1 to 5. At follow-up, students scored, on average, 3.28 on this 
scale, indicating a moderating change in interesting in these activities. 40 

Engagement in school STEM activities 

Students' level of engagement in STEM activities was measured by whether students currently 
participated in science or engineering clubs at school, fair, and camp, and whether they expressed 
interest in participating in science or engineering clubs or doing a project for a science or 
engineering fair in the future. The individual items were scored as a 0 or 1, and the composite 
indicator, which averaged cross the individual items, had a range of possible values from 0 to 1. On 
average, students' baseline score was 0.5 on this scale, suggesting a moderate level of engagement 
in school STEM activities. 

Anxiety toward science 

A student's anxiety toward science represents the amount of negative feelings that arise when a 
student thinks about science. This was measured using items about how anxious or nervous 
respondents feel when they think about science or when someone discusses science with them. 
Respondents rated statements on a Likert-type scale with four response categories ranging from 
"strongly disagree" to "strongly agree." This composite indicator had a range of possible values from 
1 to 4, and was the only indicator where higher values were less desirable (i.e., higher values 
represented greater anxiety toward science). The average baseline score on this construct was 1.58, 
reflecting that applicants generally did not feel anxious about STEM. 

Self-confidence in science 

A student's self-confidence in science represents his or her academic self-efficacy in this area (i.e., 
the sense that they can master the material or skills). This was measured using items regarding how 
students perceive their own understanding of science, as well as how easily students lose 


This number is not directly comparable with the other figures in this section, as this indicator was only 
measured at follow-up. All other figures in this chapter correspond to baseline values. 
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confidence when they have trouble understanding their science schoolwork. Each item was 
measured on a Likert-type scale with four response categories ranging from "strongly disagree" to 
"strongly agree." This composite indicator had a range of possible values from 1 to 4. On average, 
students scored 2.75 at baseline, reflecting somewhat positive self-efficacy in STEM. 

Interest in high school STEM coursework 

Students' interest in high school STEM coursework was measured by their intention to enroll in 
STEM-related courses in high school. Respondents rated their likelihood of taking classes in 13 
STEM-related subjects: earth science, life science, physical science, biology, chemistry, physics, 
anatomy and physiology, computer science, algebra, geometry, statistics, calculus and astronomy. 
Each item was measured on a Likert-type scale with four response categories ranging from "I would 
definitely not take this in high school" to "I would definitely take this in high school." This composite 
indicator had a range of possible values from 0 to 4. On average, students scored 2.67 at baseline, 
reflecting a general inclination toward taking STEM-related courses in high school. 

Interest in college STEM coursework 

Students' interest in college STEM coursework was measured by their intention to enroll in STEM- 
related courses in college. Respondents rated their interest in taking courses in eight STEM-related 
areas while in college: astronomy, biology, chemistry, computer science, engineering, math, 
medicine and physics. Each item was rated on a Likert-type scale with four response categories 
ranging from "I would definitely not take this in college" to "I would definitely take this in college." 
This composite indicator had a range of possible values from 0 to 4. On average, students scored 
2.84 at baseline, reflecting a general inclination toward taking STEM-related courses in college, 
similar to their intentions for high school. 

Interest in STEM careers 

Students' interest in STEM careers was measured by their perceived potential enjoyment of eight 
STEM-related careers: scientist, college professor, engineer, computer scientist, math teacher, 
science teacher, doctor or astronaut. Each item was rated on a Likert-type scale with four response 
categories ranging from "I would really not like this job" to "I would really like this job." This 
composite indicator had a range of possible values from 0 to 4, with students scoring 2.4 at baseline, 
on average. 

Value of science to society 

A student's sense of the value of science to society measures how much global importance a 
student associates with science, or the strength of a student's perceived external motivation to 
study science. This indicator was measured using items regarding how respondents view the utility 
of science and the importance of learning science in today's world. Each item was measured on a 
Likert-type scale with four response categories ranging from "strongly disagree" to "strongly agree." 
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This composite indicator had a range of possible values from 1 to 4. The average student score at 
baseline on this composite was 3.28, reflecting general agreement that science is of value to society. 

While scores varied by indicator, baseline levels on almost all indicators were notably high. Overall, 
SEMAA applicants are engaged and interested in science even before they participate in a SEMAA 
session. These applicants, however, are those that applied to the sites that implement the Saturday 
model. It is likely that sites implementing the in-school model, and possibly the after-school model, 
would enroll a set of students with wider variation in these outcomes of interest. 

Baseline Levels of Parental Outcomes 

Our final research question looked at parental perceptions of the importance of STEM and STEM 
activities, and their support for their children's engagement in these activities. Their baseline values 
on these outcomes are summarized in Exhibit 11 and described below. Similar to the student 
measures, parent baseline measures were also quite high across the board, with slightly lower 
values for parent support for students' out-of-school science activities than for the other categories. 
This may imply higher support for formal science learning than informal science learning among 
parents. Overall, the baseline measures indicate that parents had strong levels of support for STEM 
and STEM education, both present and future. 


Exhibit 11 : Parent Baseline Responses 

Overall 

(n=662) 

Outcome 

Values 

Mean 

sd 

Parent support for students’ prior out-of-school science 
activities 

0-5 

2.21 

1.12 

Parent support for students’ future out-of-school science 
activities 

0-5 

2.66 

1.01 

Parent encouragement of high school STEM coursework 

1-4 

3.60 

0.44 

Parent perception of student's need to understand 
science/math as an adult 

1-4 

3.55 

0.56 

Parent encouragement of STEM fields in college 

1-4 

3.52 

0.47 


Parental support for students' prior out-of-school science activities 

Parental support for students' out-of-school science activities was measured as the average of how 
frequently parents engaged in a series of STEM activities with their children over a three-month 
period. 41 This indicator is analogous to the corresponding student indicator measuring frequency of 
participation in informal science activities. Parents rated their frequency of involvement in the same 
10 activities, using a six-point scale of "not at all," "one time," "about twice a month," "about once a 


At baseline, parents responded with how often they had engaged in these activities over the summer. At 
follow-up, they were asked how often they had engaged in the same activities over the last three months. 
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week" or "more than once a week." This composite indicator had a range of possible values from 0 
to 5. Across the 10 activities, on average at baseline, parents scored 2.21. 

Parental support for student's future out-of-school science activities 

Future parental support for students' out-of-school science activities was measured as the average 
of how frequently parents plan to engage in these activities with their children over the three- 
month period following the administration of each survey. Parents rated how frequently they plan 
to participate in the same 10 activities that they rated for past participation, using the same six- 
point scale used to describe the frequency with which they had previously engaged in these 
activities. This composite indicator had a range of possible values from 0 to 5, and parents scored, 
on average, 2.66 at baseline, suggesting they planned to engage in STEM activities more frequently 
than they had in the past three months. 

Parental encouragement of high school STEM coursework 

Parental encouragement of high school STEM coursework was measured as the average of how 
likely parents would be to encourage their children to take a series of STEM-related classes in high 
school. Respondents rated their likelihood of encouraging their child to take classes in 15 STEM- 
related areas: physics, biology, chemistry, earth science, astronomy, computer science, anatomy 
and physiology, advanced physics, advanced biology, advanced chemistry, algebra, geometry, 
trigonometry, calculus and statistics. This composite indicator had a range of possible values from 1 
to 4, ranging from "would definitely not encourage" to "would definitely encourage." On average, 
parents scored 3.6 on this construct at baseline, suggesting a high level of encouragement for 
students to take STEM classes in high school. 

Parental perception of student's need to understand science/math as adult 

Parental perception of students' need to understand science and mathematics as an adult was 
measured as an average of four items, asking parents if their children would need to have a good 
understanding of basic and advanced science and mathematics as an adult. Parents rated each item 
on a Likert-type scale with four response categories ranging from "strongly disagree" to "strongly 
agree." This composite indicator had a range of possible values from 1 to 4, and parents scored an 
average of 3.5 at baseline, suggesting that parents generally perceived a strong need for their 
children to understand science and math as adults. 

Parental encouragement of STEM fields in college 

Parental encouragement of pursuing STEM fields in college was measured by how likely parents 
would be to encourage their children to earn a college degree in a variety of fields. Respondents 
rated their likelihood of encouraging their child to major in eight STEM-related areas: physics, 
biology, computer science, chemistry, engineering, mathematics, medicine and astronomy. 
Respondents rated each item on a Likert-type scale with four response categories ranging from 
"would definitely not encourage" to "would definitely encourage." This composite indicator had a 
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range of possible values from 1 to 4, and parents had an average score of 3.52, reflecting a high level 
of encouragement by parents for students to pursue STEM coursework. 

Overall, baseline levels on the outcomes that were investigated in the impact study showed an 
applicant pool that entered the SEMAA program with high levels of interest in science and parental 
support for their interest and engagement in STEM. 
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Chapter 5: Outcomes of SEMAA 


In this chapter, we present findings that address the research questions about the effect of SEMAA 
on the interests and attitudes of students and parents. First, we provide a brief summary of analysis 
results. Then, in the sections that follow we present the results from models that test the overall 
effect of the program— across both previous participants and new participants— followed by a 
presentation of the results of models that tested whether the impacts of SEMAA differed for 
students with prior experience in the SEMAA program. Finally, we present results from exploratory 
analyses that investigated the relationship between the level of participation in SEMAA classes 
during the current session and the outcomes being investigated. 

The study found no overall impacts on the measures of students' interest, attitudes, or participation 
in STEM activities, and no significant impacts on parent support or encouragement of STEM 
activities. Analysis of the same outcomes measured at baseline indicated that in both treatment and 
control groups, students' interest, attitudes and participation, and parents' support and 
encouragement were high. In both the treatment and control groups, student interest, attitudes, 
participation in STEM activities tended to increase from the baseline to follow-up, but the increase 
was not significantly greater for the treatment group. 

Although no overall effects of the program were found, SEMAA did have an impact among returning 
students on their frequency of participation in informal science activities. This finding is consistent 
with the longitudinal structure of SEMAA, which provides new learning experiences for students in 
each succeeding grade, and with the program theory that SEMAA has a cumulative effect on 
participants. Further, exploratory analyses suggest that students who attend more SEMAA classes 
within a session have higher levels on the outcomes of interest at the end of the session. Similarly, 
parent attendance in more Family Cafe sessions in positively related to some outcomes of interest. 

Overall SEMAA Impacts 

The first series of analyses tested for an overall impact of the SEMAA program on student and 
parent outcomes of interest. Notably, the study found no impacts of SEMAA on participants. Exhibit 
12 presents the specific findings for each outcome measure. The means and standard deviations of 
the outcome measures for the control group are presented, followed by the treatment group mean. 
The treatment effect is the estimated effect of SEMAA, in the units of the original scale for each 
measure. In comparison both to the overall range of values and the control group means and 
standard deviations, the magnitudes of the estimated treatment effects were all practically small, 
and statistically indistinguishable from zero. 

Not all individuals who were assigned to the treatment group actually attended the SEMAA program 
sessions; 19 percent of the individuals in the treatment group attended zero sessions. Therefore, an 
exploration of the relationship between outcomes and number of sessions attended is presented 
later in this chapter. 


Abt Associates Inc. 


National Evaluation of SEMAA 


51 




National Evaluation of SEMAA Abt Associates Inc. 


Exhibit 12: Impact Estimates of SEMAA (n=396) 


Outcome and (Range of Values) 


Control 

Standard 

Deviation 


Estimated 
Treatment 
Effect c 



p-value 

Control 
Mean a 

Treatment 
Mean b 

Standard 

Error 

Effect 
Size d 

Student Outcomes 


Desire to do science (1-4) 

3.22 

0.50 

3.20 

-0.03 

0.05 

-0.05 

0.574 

Frequency of participation in informal science activities (0-5) 

2.53 

1.08 

2.68 

0.14 

0.10 

0.13 

0.161 

Interest in informal science activities (0-4) 

3.35 

0.50 

3.34 

-0.01 

0.05 

-0.02 

0.867 

Change in interest in informal science activities f (1-5) 

3.28 

0.79 

3.36 

0.08 

0.09 

0.10 

0.364 

Engagement in STEM activities (0-3) 

0.49 

0.30 

0.51 

0.01 

0.03 

0.04 

0.705 

Anxiety toward science 9 (1-4) 

1.49 

0.50 

1.53 

0.04 

0.04 

0.08 

0.371 

Self-confidence in science (1-4) 

2.78 

0.30 

2.81 

0.04 

0.03 

0.12 

0.215 

Interest in high school STEM coursework (0-4) 

2.66 

0.88 

2.66 

0.00 

0.09 

0.00 

0.970 

Interest in college STEM coursework (0-4) 

2.74 

0.88 

2.85 

0.12 

0.09 

0.13 

0.195 

Interest in STEM careers (0-4) 

2.42 

0.84 

2.46 

0.04 

0.08 

0.05 

0.588 

Value of science to society (1-4) 

3.35 

0.44 

3.33 

-0.01 

0.04 

-0.03 

0.744 

Parent Outcomes 

Parent support for out-of-school science activities, last 3 months (0-5) 

2.38 

0.96 

2.47 

0.08 

0.08 

0.09 

0.315 

Parent support for out-of-school science activities, next 3 months (0-5) 

2.60 

0.94 

2.63 

0.03 

0.08 

0.03 

0.721 

Parent encouragement of high school STEM coursework (1-4) 

3.51 

0.46 

3.55 

0.04 

0.04 

0.09 

0.337 

Parent perception of student’s need to understand science and math as 

3.57 

0.52 

3.52 

-0.04 

0.06 

-0.09 

0.421 

an adult (1-4) 








Parent encouragement of STEM fields in college (1-4) 

3.47 

0.47 

3.49 

0.01 

0.04 

0.03 

0.747 


Notes: 


Fitted models to estimate the effect of SEMAA for student outcomes included controls for baseline measure of the outcome variable, whether student had previously participated in SEMAA, grade 
level, whether the student knows any family member working in a STEM field, parent’s highest education completed, whether family is high-income, whether parent works for NASA, amount of help 
student received from adult in completing baseline survey, and whether the student uses only English in school. For parent outcomes, controls included: baseline measure, whether student had 
previously participated in SEMAA, parent’s highest education completed, whether family is high-income, whether parent works for NASA, and whether parent works in a STEM field. All fitted 
models were multilevel models, and included random intercepts for SEMAA sites. 

a Control means and standard deviations are control group unadjusted means and standard deviations from the follow-up measure of the outcome. 
b Treatment means are adjusted means from the fitted model, and are calculated by adding the unadjusted control group mean to the estimated treatment effect. 
c The estimated treatment effect is the estimated effect of SEMAA on the outcome from the fitted model. 

d The effect size is the estimated treatment effect converted to standard deviation units; it is a common calculation that allows for comparisons of effect sizes across different measures and research 
studies. 

f The change in interest in informal science activities outcomes had no baseline measure. 

g In the scale of the anxiety in science outcome, lower values represent more positive outcomes (less anxiety in science). 

Exhibit reads: On the “desire to do science” outcome, the observed control group mean and standard deviation at follow-up were 3.22 and 050. The estimated mean for the treatment mean was 3.20, 
The estimated impact of SEMAA was -0.03 (or -0.03 standard deviations), which is not statistically significant (p=0.574). 





As discussed in Chapter Four, both students and parents entered the study with relatively high 
interests and attitudes. Since the students and parents interested in and willing to apply to the 
program appear to have already been highly motivated, there was limited room for improvement in 
these measures. Nonetheless, the measures of student interest and attitudes moved in a positive 
direction between baseline and follow-up for the treatment group students, and there were 
statistically significant baseline to follow-up gains in the measures of frequency in participation of 
informal science activities, number of informal science activities, self-confidence in science, and 
value of science to society. However, the control group students also showed baseline to follow-up 
improvements. Thus, there were no significant impacts of treatment because the improvement in 
the treatment group was not significantly different from the improvement in the control group. 
Therefore, we cannot conclude that the improvements in the treatment group's interest and 
attitudes toward STEM are greater than what they would have experienced in the absence of the 
program. It is worth noting that in another model of SEMAA that may not necessarily rely on 
recruiting highly motivated students, particularly the in-school model, baseline values for the 
outcomes of interest may be lower, allowing greater room for the program to exert an influence. 
However, it may be equally likely that SEMAA helps maintain the high levels of interest and 
motivation exhibited by participants in the Saturday model. 

Impacts for Prior SEMAA Participants and First-time Participants 

An important component of this evaluation was to test the program theory that SEMAA is a 
longitudinal program that has a cumulative effect on participants over time. Therefore, an additional 
research question for each outcome was whether SEMAA had differential effects for students with 
prior experience in SEMAA and those in the program for the first time. Tests for differential impacts 
of treatment were conducted for the student and parent outcomes. 42 For the measure of frequency 
of participation in informal science activities, the effect of SEMAA participation was different among 
students with prior SEMAA participation than it was among those without prior SEMAA participation 
(p=0.049). Specifically, among students with prior SEMAA participation, there was a significant and 
positive effect of SEMAA on the frequency of participation in informal science activities 
(estimate=0.31, p=0.019). There was not a significant treatment effect among first time SEMAA 
participants. This finding suggests that cumulative exposure to SEMAA over time is more important 
in producing change within student behaviors on this outcome than the first-time experience. 

Exploration of Variation in Participation 

In this study, students who were randomly assigned to the treatment group had the opportunity to 
participate in the Fall 2009 SEMAA session, while those in the control group were not accepted in 
the Fall 2009 session. Under the random assignment research design, the estimate of the impact of 
SEMAA, presented above, compares the outcomes for students assigned to those two groups. 


42 Tests for differential impacts were conducted on 11 outcomes for the student sample, and 5 outcomes for 
the parent sample. Using a p<0.05 criterion for statistical significance, 1 in 20 tests would be expected to 
be significant by chance. 
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However, within the treatment group there were students who did not participate in any SEMAA 
classes ("no-shows"); conversely, there were also students assigned to the control group who did 
attend SEMAA ("crossovers"). Since the presence of these groups can reduce the estimated impact 
of the program, 43 we conducted some exploratory analyses to investigate the relationship between 
the level of participation in a SEMAA session and the outcomes of interest. 44 

Exhibit 13 displays information on program participation by treatment and control group, and the 
number of sessions attended by treatment students; it reveals the presence of both no-shows and 
crossovers in the study. Specifically, nearly one-fifth of students assigned to the treatment group did 
not attend any SEMAA sessions, while over one-tenth of students assigned to the control group 
attended at least one session. 

We undertook a series of exploratory analyses to examine participation and program attendance. 
First, we looked for possible relationships between program participation and baseline measures of 
interests and attitudes, to determine whether students with greater baseline interest or motivation 
were more likely to attend more sessions. There were no significant associations between baseline 
measures and participation. We also found no relationship between baseline measures and the 
number of sessions attended, by the treatment group. Both of these results suggest that, among 
students offered the opportunity to attend the SEMAA session, participation and attendance were 
not driven by baseline interests and attitudes. 


43 For example, the score of treatment students who did not participate in the program might have reduced 
the treatment group average on each outcome at follow-up, or the if the program had a positive effect 
among control students who attended the program, the control group average on each outcome at 
follow-up would be inflated. The combination of these factors, then, produces a smaller difference in 
outcomes between the treatment and control groups, which is equivalent to a reduction in the estimated 
impact of the program. 

44 We estimated the effects of the program among those in the treatment group who actually participated 
in SEMAA by adjusting the impact estimates, under the plausible assumption that SEMAA had no impact 
on the no-shows. While this adjustment alters the magnitude of the original impact estimate, it does not 
change the statistical significance of findings (Bloom, 2005). These adjusted estimates were not 
qualitatively different from the original estimates. 
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Exhibit 13: Participation and Attendance in Fall 2009 SEMAA 

Participation in Fall 2009 SEMAA 
Sample 

Number Participated 

Percent Participated 

Full Sample (n=396) 

213 

54% 

Treatment group (n=240) 

194 

81 

Control group (n=156) 

17 

11 

Number of Fall 2009 Saturday SEMAA Sessions Attended, Treatment Group (n=240) 

Number of Saturday Sessions 



Attended 

Number of Students 

Percent of Treatment Group 

0 

46 

19.2% 

1 

2 

0.8 

2 

1 

0.4 

3 

2 

0.8 

4 

5 

2.1 

5 

17 

7.1 

6 

31 

13.0 

7 

59 

24.6 

8 

72 

30.0 

Unreported 

5 

2.1 

Note: The treatment group consisted of those students randomly assigned to the group that was offered the opportunity to 

participate in Fall 2009 SEMAA program. 




While the presence of crossovers was potentially problematic in evaluating the impact of SEMAA, as 
discussed above, they have no relevance to future program implementation. No-shows, on the 
other hand, could be an area of potential program improvement. That is, keeping students engaged 
in the program, and making sure that interested students are attending, is a program 
implementation issue that can be addressed. As demonstrated above, students who were offered 
an opportunity to attend SEMAA but never participated were not different in terms of baseline 
interests and attitudes from students who did participate. 

An additional exploratory analysis was conducted to determine whether there were associations 
between the number of SEMAA sessions attended and outcomes, among students in the treatment 
group. Exhibit 14 presents the results of this analysis, and indicates positive and significant 
relationships between number of sessions attended and six of the outcomes of interest: desire to do 
science, change in interest in informal science activities, anxiety toward science, interest in high 
school STEM coursework, interest in college STEM coursework, and parent encouragement of 
pursuing STEM fields in college. 
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Exhibit 14: Associations between Attendance and Outcomes, Within Treatment Group 
(n=240) 



Estimated 



Outcome and (Range of Values) 

Effect a 

Standard Error 

p-value 

Student Outcomes 

Desire to do science (1-4) 

0.03* 

0.01 

0.013 

Frequency of participation in informal 
science activities (0-5) 

0.04 

0.02 

0.081 

Interest in informal science activities (0-4) 

0.01 

0.01 

0.256 

Change in interest in informal science 
activities (1-5) 

0.04* 

0.02 

0.023 

Engagement in STEM activities (0-3) 

0.01 

0.01 

0.174 

Anxiety toward science (1-4) 

-0.03** 

0.01 

0.002 

Self-confidence in science (1-4) 

0.01 

0.01 

0.227 

Interest in high school STEM coursework 
(0-4) 

0.04* 

0.02 

0.048 

Interest in college STEM coursework (0- 
4) 

Interest in STEM careers (0-4) 

0.04* 

0.02 

0.050 

0.01 

0.02 

0.376 

Value of science to society (1-4) 

0.01 

0.01 

0.284 

Parent Outcomes 

Parent support for out-of-school science 
activities, last 3 months (0-5) 

0.01 

0.02 

0.721 

Parent support for out-of-school science 
activities, next 3 months (0-5) 

0.00 

0.02 

0.872 

Parent encouragement of high school 
STEM coursework (1-4) 

0.01 

0.01 

0.143 

Parent perception of student’s need to 
understand science and math as an adult 

0.01 

0.01 

0.614 

(1-4) 

Parent encouragement of STEM fields in 

0.02* 

0.01 

0.048 


college (1-4) 

Notes: 

* p < 0.05, **p<0.01 

Attendance, or number of Saturday SEMAA sessions attended, was from parent report on the follow-up survey. 

a Estimated effect of one additional Saturday SEMAA session attended. Students offered the opportunity to participate 
in the Fall 2009 SEMAA program could have attended up to 8 Saturday sessions. 

Exhibit Reads: Each additional session of SEMAA was related to a difference of 0.03 on the outcome “desire to do 
science”. The estimated effect was statistically significant (p= 0.013) 


The reported estimates indicate the estimated effect of one additional session attended on each 
outcome. If we were to consider the most extreme disparities in attendance, between students who 
attended zero sessions and students who attended all eight sessions, the comparable effect sizes 
would be quite large, on the order of one-third to one-half a standard deviation, except for interest 
in high school STEM coursework, which would indicate about one full standard deviation difference 
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in the outcome at follow-up, 45 even after controlling for the baseline measures of outcomes, 
demographic characteristics, and prior SEMAA participation. 

A final exploratory analysis was conducted to investigate whether there was a relationship between 
the number of Family Cafe sessions attended by parents and any of the parent or student outcomes 
among the treatment group. In the treatment group, 65 percent of parents reported attending at 
least one Family Cafe session; the mean number attended was near three. Exhibit 15 presents the 
results of the analysis that investigated the relationship between number of Family Cafe sessions 
attended and the outcomes investigated. The number of Family Cafe sessions attended by the 
parent was positively related to parent participation in informal science activities with their child. 
Among student outcomes, parent attendance was positively related to desire to do science, change 
in interest in informal science activities, lower anxiety toward science, and interest in collegiate 
STEM majors/coursework. 46 

These results indicate several positive associations between the number of sessions attended and 
outcomes at follow-up levels of interest and attitudes regarding STEM. There were also positive 
relationships between the number of Family Cafe sessions attended and outcomes at follow-up. 
Although the results of these exploratory analyses cannot be interpreted as program impacts, they 
suggest potentially important factors to consider in ongoing implementation and planning for future 
program implementation. 


45 Except for interest in high school STEM coursework, in which there was about a full standard deviation 
difference in the outcome. 

46 When both student and parent participation were tested as predictors simultaneously, the number of 
Family Cafe sessions attended by the parent was positively related to student desire to do science and 
parent participation in informal science activities with their child over the last three months. The number 
of Saturday SEMAA sessions attended by the student was positively associated with lower anxiety toward 
science. 
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Exhibit 15: Associations between Family Cafe Attendance and Outcomes, Within Treatment 
Group (n=240) 



Estimated 



Outcome and (Range of Values) 

Effect a 

Standard Error 

p-value 

Student Outcomes 

Desire to do science (1-4) 

0.04** 

0.01 

0.002 

Frequency of participation in informal 
science activities (0-5) 

0.04 

0.02 

0.097 

Interest in informal science activities (0-4) 

0.01 

0.01 

0.258 

Change in interest in informal science 
activities (1-5) 

0.05 

0.02 

0.024 

Engagement in STEM activities (0-3) 

0.00 

0.01 

0.756 

Anxiety toward science (1-4) 

-0.03* 

0.01 

0.014 

Self-confidence in science (1-4) 

0.01 

0.01 

0.115 

Interest in high school STEM coursework 

0.03 

0.02 

0.095 

(0-4) 

Interest in college STEM coursework 
(0-4) 

0.05* 

0.02 

0.018 

Interest in STEM careers (0-4) 

0.01 

0.02 

0.608 

Value of science to society (1-4) 

0.00 

0.01 

0.985 

Parent Outcomes 

Parent support for out-of-school science 
activities, last 3 months (0-5) 

0.04* 

0.02 

0.031 

Parent support for out-of-school science 
activities, next 3 months (0-5) 

0.03 

0.02 

0.149 

Parent encouragement of high school 
STEM coursework (1-4) 

0.01 

0.01 

0.283 

Parent perception of student’s need to 
understand science and math as an adult 

0.00 

0.01 

0.910 

(1-4) 

Parent encouragement of STEM fields in 
college (1-4) 

0.00 

0.01 

0.673 


Notes: 

* p < 0.05, **p<0.01 


Exhibit Reads: Each additional Family Cafe session was related to a difference of 0.04 on the outcome “desire to do 
science.” The estimated effect was statistically significant (p=0.001). 
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Chapter 6: Conclusions and Implications 


In this study, we examined the implementation of SEMAA across sites and investigated the 
relationship between outcomes of interest (such as desire to study science and pursue a career in 
STEM) and participation in the SEMAA Saturday program. As outlined in the logic model that depicts 
SEMAA's theory of change (see Exhibit 4), we focused on short-term outcomes that could be 
measured in the time available for evaluation activities. The implementation component provided 
insights into the evolution of SEMAA over time, highlighted important barriers to 
institutionalization, and yielded recommendations for future directions. The impact component 
used a rigorous methodology to investigate the impact of program participation on 10 student and 5 
parent short-term outcomes of interest. 

Summarized in detail below, results of the impact analysis are mixed, providing limited support for 
the efficacy of the SEMAA program. The study found no overall impacts of the eight-week SEMAA 
Saturday program, but yielded two important findings: (1) increased participation in informal 
science activities among repeat participants, and (2) a positive relationship between student 
attendance and several outcomes of interest. 

Given these mixed results, a natural question is why the study did not detect stronger effects. A 
likely explanation involves the self-selected nature of the population that is attracted to the 
Saturday program— namely, students from families sufficiently interested in STEM to seek out or 
respond to recruitment efforts for this STEM enrichment opportunity, and to do so repeatedly. 
(Recall that about 40 percent of the students in our sample were repeat participants that by virtue 
of their reenrollment demonstrate high interest.) This is confirmed by the high share of SEMAA 
parents in the STEM workforce (33 percent). Not surprisingly, upon enrollment, these families 
already display high average levels on several outcomes (see Exhibits 10 and 11), making it difficult 
to detect increases in these areas. In these cases the program may hope to maintain this high level 
of interest over the long term. 

This evaluation has allowed us to see the (often high) estimated measures at the time of enrollment. 
We were also able to confirm strong recurring participation in SEMAA, and reveal the important role 
that an in-school model may play in SEMAA's future. As such, we urge the program to consider, as 
recommended below, tracking participants and nonparticipants across models (not just Saturday 
programs) to observe students' educational trajectories and measure their long-term educational 
outcomes. 

Conclusions 

1. Results did not reveal overall impacts of the SEMAA program on short-term outcomes, but 
among prior participants, SEMAA led to increased participation in informal science activities. The 

study found high levels of interest and engagement at baseline, and no overall effect of the Saturday 
SEMAA model on short-term outcomes. However, the program did find an impact on participation in 
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informal science activities among individuals who had previously enrolled in the program. This 
impact of SEMAA on prior participants aligns with SEMAA's longitudinal approach to engage 
students by offering STEM opportunities across multiple grades. It also signals increased student 
engagement in science, which the SEMAA model links to increased future participation in STEM 
education. 

2. Student and parent attendance had a positive relationship with several outcomes of interest. 

Exploratory analyses indicate that the number of Saturday classes students attended within a 
session was related to a greater (a) desire to engage in science-related activities, (b) interest in high 
school STEM courses, (c) interest in college STEM coursework among students, and (d) parental 
encouragement for students to pursue a STEM field in college; to a perception that (e) their interest 
in informal science activities had increased; and finally to (f) lessened anxiety toward science. These 
findings suggest that increased exposure to the SEMAA program may be associated with short-term 
outcomes that are important precursors to subsequent enrollment in a STEM baccalaureate 
program. In addition, parent participation in Family Cafe session was positively related to (a) parent 
participation in informal science activities with their child, (b) child's desire to do science, (c) a 
change in their child's interest in informal science activities, (d) child's lower anxiety toward science, 
and (e) their child's interest in collegiate STEM majors/coursework. 

3. Sites report achieving continued student participation overtime. Sites implementing out-of- 
school models report that more than half of participants continue in SEMAA from one year to the 
next . 47 In sites implementing the in-school model, participation is mandatory for all enrolled 
students and attrition is driven by school mobility. This pattern of reenrollment (out-of-school) or 
continued exposure (in-school) is (a) consistent with SEMAA's program theory that students should 
be engaged repeatedly to maintain and further develop an interest in and engagement with science, 
and (b) linked to the SEMAA and NASA Office of Education goal of attracting and retaining students. 

4. In the short term, three models of implementation coexist; the Saturday model is the most 
prevalent, but the in-school model is being used or considered to sustain projects in the long term. 

The SEMAA project is characterized by three models of implementation— two conducted out of 
school hours (Saturday model and after-school model) and one during school hours (in-school 
model). NASA encourages the Saturday model, but site selection of an implementation model 
responds to the needs of the particular site. Sites in rural locations favor the Saturday or in-school 
over the after-school model. Those in urban settings with greater student demand, though generally 
starting out as Saturday programs, mentioned shifting to the in-school program to serve more 
students at a lower cost. 

At some sites, the in-school model— which does not need to cover teacher salaries and therefore 
requires less funding than the out-of-school programs— is seen by site directors as a viable solution 


47 This estimate is higher than the repeat participation observed among sampled students in part because 
programs were asked to stop recruiting students six weeks prior to the beginning of the session in order 
to randomly assign applicants into treatment and control groups for the study. 
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to the need to raise funds to sustain the project after NASA funding ends or to expand the project in 
response to increased demand, as evidenced by projects that cited overcoming funding constraints 
by transitioning to in-school. One potential drawback of in-school programs is that, over time, the 
content or science/NASA focus might be neglected or deemphasized as priority is given to NCLB- 
tested subjects and topics. But these programs— assuming checks are in place to ensure the 
integrity of content and pedagogy— may reach greater numbers of students and facilitate the 
institutionalization of the SEMAA program. 

5. Site directors have not assumed responsibility for fundraising. In discussing fundraising, site 
directors offered words of resistance. Most of them are either inexperienced with fundraising or 
unable to carry out this task given multiple other responsibilities that compete for their time. Site 
directors have developed partnerships and collaborations, but none have raised sufficient funds to 
sustain the project as an out-of-school program. 

6. Limited student monitoring makes investigating long-term outcomes difficult. SEMAA's program 
theory posits that participation will have effects that extend well beyond the short-term outcomes 
investigated in this study. Some of these long-term outcomes are increased enrollment in high 
school mathematics and science courses, participation in post-secondary STEM programs of studies, 
and employment in STEM. But programs currently do not collect the data necessary to assess the 
educational and career trajectories of participants. 

Recommendations 

1. Continue the existing emphasis on repeat participation. SEMAA's longitudinal approach to 
engaging students aligns with NASA's education framework, which is modeled as continued and 
increasing involvement with STEM. Although the in-school model has attendance within a session 
and repeat participation from year to year built into it by virtue of the model itself, in the out-of- 
school models attendance and continued participation is not mandatory. Given the effect of SEMAA 
on individuals who have previously participated in the program, ongoing project efforts to foster 
repeat participation should be encouraged. 

2. Encourage strategic planning. The in-school model, though perceived by project directors as a 
viable alternative, lacks some positive attributes that are characteristic of the Saturday and after- 
school models— namely, these are "special" programs with a clear science focus that form a 
community of learners by engaging the students and their families. Project sites should be 
encouraged to contemplate the alternatives and engage in strategic planning early on, so the 
transition to an in-school model, if carried out, is the result of a conscious and careful plan and not a 
last-minute response to lack of funding as was the case in some sites. That careful plan should 
include periodic checks of adherence to the SEMAA content and pedagogy. Further, if the Saturday 
model fits the needs of the population being served at a site, early strategic planning may allow the 
Saturday model to be a sustainable option. 
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3. Monitor the various models to understand prospects for sustainability and to prevent crowding 
out of science content. The SEMAA project may be able to help sites transition beyond NASA 
funding by gathering and sharing information about the challenges that have been faced and efforts 
that have succeeded across the current and former sites. In addition, the SEMAA project should 
ensure that in-school model sites adhere to the programmatic requirements in terms of content, 
methods and time coverage, especially given NCLB-driven pressure to focus on mathematics and 
reading. 

4. Support continued grantee training on fundraising and partnerships. On average, sites have 
raised significant support for their projects, but these efforts have not been sufficient for projects to 
become independent of NASA funding. Consequently, sites need (a) assistance in developing a 
sustainability plan based on existing capacity and mindful of local needs and constraints, and (b) 
continued training and encouragement to develop fundraising skills, as most grantees are not 
experienced in fundraising. 

5. Encourage student monitoring data collection. The immediate goal of SEMAA is to inspire, 
engage and educate K-12 students. The ultimate goal of this effort is to encourage students to 
pursue STEM fields of study in college, thereby increasing the number of students who enroll in 
STEM majors in college. Consequently, programs should be encouraged to collect information on 
participating students to measure program exposure and to facilitate future tracking. Perhaps the 
new NASA Office of Education Performance Measurement System (OEPM) could be used for 
projects to submit individual participant data that could be retrieved at a later time to follow up on 
past participants to determine whether they enrolled in college and in which field. If possible, data 
on nonparticipants at the same schools should also be collected to facilitate comparisons and 
establish, through a quasi-experimental design, the success of the program. 

Future Directions for SEMAA 

There are two possible directions the SEMAA program could take; for simplicity, we call these NASA- 
dependent and NASA-independent. The NASA- dependent path would require that NASA continue 
to renew funding over time. Indeed, many sites have received funding beyond the initial award 
period. Renewed funding would have the advantage of guaranteeing project continuity, assuming 
adequate student demand, but would have the disadvantage of preventing the expansion of the 
program to new sites, unless the program budget were to grow. As it stands, the program reaches 
less than 21,000 (direct participants) of the more than 18 million minority or close to 20 million 
economically disadvantaged students enrolled in K-12 public schools throughout the nation. 48 

The other alternative is for sites to fundraise to support continuation of the project after NASA 
funding has ended. Becoming NASA-independent reflects current program expectations. The 
evidence uncovered in this study suggests that sites have been unable to raise the funds necessary 


Figures correspond to the 2007-2008 school year. Poverty is measured as qualifying for free or reduced 
lunch (NCES, 2009b). 
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to outlive NASA funding as an out-of-school program. In fact, in order to continue offering services 
post-NASA funding, some sites have chosen (or are planning) to transition to an in-school program, 
which is much less expensive than an out-of-school program (whether Saturday or after-school) and 
avoids transportation barriers. We learned from sites that the national SEMAA office, through 
contractor Paragon TEC, was planning to offer training on fundraising. The national SEMAA office 
subsequently confirmed that such training has been offered to sites. Welcomed by site directors, 
this training may be just what is needed to assist grantees in continuing the program post-NASA 
funding. It will be important to monitor site progress in fundraising post-training to assess its 
effectiveness. 


Abt Associates Inc. 


National Evaluation of SEMAA 


63 




THIS PAGE INTENTIONALLY BLANK 



References 


Bandura, A. (1993). Perceived self-efficacy in cognitive development and functioning. Educational 
Psychologist, 28, 117-148. 

Bandura, A., Barbaranelli, C., Caprara, G.V., & Pastorelli, C. (2001). Self-efficacy beliefs as shapers of 
children's aspirations and career trajectories. Child Development, 72, 187-206. 

Barnett, M., Lord, C., Strauss, E., Rosea, C., Langford, H., Chavez, D., & Deni, L. (2006). Using the 
urban environment to engage youths in urban ecology field studies. The Journal of Environmental 
Education, 37, 3-11. 

Betz, N.E., & Hackett, G. (1983). The relationship of mathematics self-efficacy expectations to the 
selection of science-based college majors. Journal of Vocational Behavior, 23, 329-345. 

Bleeker, M.M., & Jacobs, J.E. (2004). Achievement in math and science: Do mothers' beliefs matter 
12 years later? Journal of Educational Psychology, 96, 97-109. 

Bloom, H. S. (ed.) (2005). Learning more from social experiments: Evolving analytic approaches. 
New York: Russell Sage Foundation. 

Bondurant, L., Davis, H., Davey, B., & Davis, R.M. (2008). SEMAA project evaluation analysis. 
Unpublished manuscript. Cleveland, OH: Paragon TEC. 

Eccles, J.S. (2005) Influence of parents' education on their children's educational attainment: The 
role of parent and children's perceptions. London Review of Education, 3, 191-204. 

Fadigan, K.A., & Hammrich, P.L. (2004). A longitudinal study of the educational and career 
trajectories of female participants of an urban informal science education program. Journal of 
Research in science Teaching, 1, 835-860. 

Farenga, S.J., & Joyce, B.A. (1998). Science-related attitudes and science course selection: A study 
of high-ability boys and girls. Roeper Review, 20, 247 

Federman, M. (2007). State graduation requirements, high school course taking, and choosing a 
technical college major. Berkeley Electronic Journal of Economic Analysis and Policy, 7, 1-32. 

Fraser, B.L. (1981). Test of science-related attitudes. Victoria, Australia: Australian Council for 
Educational Research. 

Gibson, H., & Chase, C. (2002). Longitudinal impact of an inquiry-based science program on middle 
school students' attitudes toward science. Science Education, 86(5), 693. 

Hart, B., & Risley, T.R. (1995). Meaningful differences in the everyday experiences of young American 
children. Baltimore, MD: Paul H. Brooks. 


Abt Associates Inc. 


National Evaluation of SEMAA 


65 




Hulett, L., Williams, T., Twitty, L., Turner, R., Salamo, G., & Hobson, A. (2004). Inquiry-based 
classrooms and middle school student perceptions about science and math. Paper presented at the 
2004 Annual Meeting of the American Educational Research Association, San Diego, CA. April 15, 
2004. 

Jacobs, J.E., & Bleeker, M.M. (2004). Girls' and boys' developing interests in math and science: Do 
parents matter? New Directions for Child and Adolescent Development, 106, 5-21. 

Kane, T. (2004). The impact of after-school programs: Interpreting the results of four recent 
evaluations. Working Paper. New York: William T. Grant Foundation. 

Lauer, P.A., Akiba, M., Wilkerson, S.B., Apthorp, H.S., Snow, D., & Martin-Glenn, M. (2004). The 
effectiveness of out-of-school-time strategies in assisting low-achieving students in reading and 
mathematics: A research synthesis. (Updated ed.). Aurora, CO: Mid-continent Research for 
Education and Learning. 

Lent, R.W., Brown, S.D., & Larkin, K.C. (1986). Self-efficacy in the prediction of academic 
performance and perceived career options. Journal of Counseling Psychology, 33, 265-269. 

Levine, P.B., & Zimmerman, D.J. (in press). After school programs. In P.B. Levine & D.J. Zimmerman 
(Eds.), Targeting investments in children: Fighting poverty when resources are limited. Chicago: 
University of Chicago Press. Downloaded February 7, 2010 from 
http://www.nber.org/chapters/cll725. 

Markowitz, D.G. (2004). Evaluation of the long-term impact of a university high school summer 
science program on students' interest and perceived abilities in science. Journal of Science 
Education and Technology, 13, 395- 407. 

Mau, W.C. (2003). Factors that influence persistence in science and engineering career aspiration. 
Career Development Quarterly, 51, 234-243. 

National Aeronautics and Space Agency. (2007a). NASA Education Strategic Coordination 
Framework: A portfolio approach (NP-2007-01-456-HQ). Washington, DC: Author. 

National Aeronautics and Space Agency. (2007b). NASA SEMAA 2007 Annual Report. Cleveland, OH: 
Paragon TEC. 

National Center for Education Statistics. (1997). Reading and mathematics achievement growth in 
high school (Issue Brief 98-038). Washington: U.S. Department of Education. 

National Center for Education Statistics. (2007). Advanced mathematics and science course taking in 
the spring high school senior classes of 1982, 1992, and 2004 (NCES #2007-312). Downloaded 
February 8, 2010 from http://nces.ed.gov/pubs2007/2007312.pdf. 


66 


National Evaluation of SEMAA 


Abt Associates Inc. 




National Center for Education Statistics. (2009a). Students who study science, technology, 
engineering, and mathematics (STEM) in postsecondary education (NCES #2009-161). Downloaded 
February 8, 2010 from http://nces.ed.gov/pubs2009/2009161.pdf. 

National Center for Education Statistics. (2009b). Schools and Staffing Survey (SASS), public school, 
BIE School, and private school data files, 2007-2008 (NCES 2009-321). 

National Research Council. (2008). NASA's Elementary and Secondary Education Program: Review 
and critique. Washington, DC: National Academies Press. 

National Science and Technology Council. (2008). Finding out what works: Agency efforts to 
strengthen the evaluation of federal science, technology, engineering and mathematics (STEM) 
education programs. A Report of the National Science and Technology Council Subcommittee on 
Education. Downloaded February 6, 2009 from 

http://www.ostp.gov/galleries/NSTC%20Reports/NSTC_Education_Report_Complete.pdf. 

Oakes, J. (1990). Opportunities, achievement, and choice: Women and minority students in science 
and mathematics. Review of Research in Education, 16, 153-339. 

Rose, FI., & Betts, J.R. (2004). The effect of high school courses on earnings. The Review of 
Economics and Statistics, 86(2), 497-513. 

Sadler, P.M., & Tai, R. FH. (2007). The two high-school pillars supporting college science. Science, 

317, 457-458. 

Schiefele, U., Krapp, A., & Winteler, A. (1992). Interest as a predictor of academic achievement: A 
meta-analysis of research. In K.A. Renninger, S. FHidi, and A. Krapp (Eds.), The role of interest in 
learning and development (pp. 183-212). Mahwah, NJ: Erlbaum. 

Sells, L.W. (1980). Mathematics: The invisible filter. Engineering Education, 70, 340-341. 

Shernoff, D.J., & Floogstra, L. (2001). Continuing motivation beyond the high school classroom. New 
Directions for Child and Adolescent Development, 93, 73-87. 

Simpkins, S.D., Davis-Kean, P.E., & Eccles, J.S. (2006). Math and science motivation: A longitudinal 
examination of the links between choices and beliefs. Developmental Psychology, 42, 70-83. 

Stakes, J.E., & Mares, K.E. (2005). Evaluating the impact of science enrichment programs on 
adolescents' science motivation and confidence: The splashdown effect. Journal of Research in 
Science Teaching, 42, 359-375. 

U.S. Department of Education. (2007). Report of the Academic Competitiveness Council. Washington, 
DC. 


Abt Associates Inc. 


National Evaluation of SEMAA 


67 




Watt, Eccles, J.S., & Durik, A.M. (2006). The leaky mathematics pipeline for girls: A 

motivational analysis of high school enrollments in Australia and the USA. Equal Opportunities 
International, 25, 642-659. 

Weinburgh, M.H., & Steele, D. (2000). The modified attitude toward science inventory: Developing 
an instrument to be used with fifth grade urban students. Journal of Women and Minorities in 
Science and Engineering, 6, 87-94. 


68 


National Evaluation of SEMAA 


Abt Associates Inc. 




Appendix A: Methodological Details 


Implementation Study 

Document reviews were conducted of standard operating procedures (SOPs) that provide 
programmatic details of each site, quarterly performance reports, and project websites. This review 
became the basis of summaries of key information across sites and guided the development of the 
interview protocol. Interviews were conducted with all open sites (14) and with 6 of the 11 closed 
sites. 

In preparation for the interviews, senior staff conducted a telephone interview training session for 
junior members of the evaluation team. Subsequently, evaluation team members reviewed the 
project files and project websites to generate a project overview sheet outlining key statistics and a 
description of project components and activities. Interviews with active sites on average lasted 
about an hour, while those with closed sites generally lasted about 20 minutes. With permission of 
the interviewee, interviews were tape recorded and later transcribed verbatim. A Microsoft Access 
database was created so that interview data could be entered, stored, and retrieved for data 
analysis. 

Note that, to circumvent influential outliers, in some instances we opted to focus on the median 
rather than the mean in reporting group average tendencies. Given the small sample sizes (N=14 
active and 6 closed sites), outliers in both directions may skew group averages significantly. For 
example, we report estimates of individuals served through AEL outreach activities. The range 
reported by the active sites is from 0 to 5,500, with a mean of 1,165 but a median of 775. In this 
instance the maximum value pulls the average estimate upward. Consequently, the median, which 
provides the midpoint in the distribution, provides a more stable and informative estimate. 

We also note that, for statistics reported for active sites, we technically count with a complete 
census and, therefore, accurate information. But some uncertainty or "noise" in the data is 
introduced by the means of data collection— namely, telephone interviews that require respondents 
to recall facts from memory and encourage rough estimation if exact numbers are unavailable. In a 
few instances where respondents could not recall some facts or statistics, we relied on the quarterly 
reports submitted to NASA. 
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Exhibit A.1 : Site Interviews Conducted 

v' = Interviews 

Open SEEMA Sites 

Albany State University 

✓ 

Cuyahoga Community College 


Fernbank Science Center 


Martinsville City Schools 


Miami-Dade County Schools 


Morgan State University 


New Mexico State University 


Oglala Lakota College 


Richland County School District One 


SECME/Tennessee State University 


University of District of Columbia 


Warren County High School 


Wayne State University 


York College/CUNY 


Closed SEEMA Sites 

Central Arizona College 


Christian Fenger Academy High School 


Lakeland Community College 

s 

Livingston College 


Medgar Evers College/CUNY 


Sinclair Community College 


St. Louis Public Schools 


University of Maryland Eastern Shore 

s 

Wilberforce University 

s 

Wiley College 

s 

Winston-Salem State University 

s 


Impact Study 

Approach to Estimating Impacts 

The treatment impacts were estimated in two-level models with students (level-1) nested in sites 
(level-2). The models have random intercept terms for sites, and estimate a single, combined 
treatment effect that represents the average treatment across sites. The model used to test 
student/parent impacts can be written in hierarchical form, as shown below. 

The level-1, or student (parent)-level model is: 

n 

Y ij = Poj + P\ j(trt tj ) + P 2 ](P r e l i - pre) + ^ kj (cov ar^ - cov cir) + £ij 

k = 3 

The Level-2, or site-level model is: 


Po j - roo + «oi 
Plj = /io 
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Pkj ~ YkO 


where the level-1 factors are defined as: 

Y.. = the outcome measure of the f h student/parent in the /'site, at the end of 

v 

the Fall 2009 SEEMA session; 

trtjj = an indicator variable denoting whether the i th student/parent in the /' site is 

a treatment student/parent ( trt u = 1) or a control student/parent ( trt„ = 0); 

u u 

pre - - pre = the baseline outcome measure for the i' h student/parent in the /' site, 

centered at the grand mean; 

covar, -co var = covariate measure of the i th student//parent in the /' site, centered at the 
grand mean; 

£.. = the student/parent -level residual of the i' h student/parent in the /' site. The 

assumed distribution of these residuals is normal, with mean = 0, and variance = <j 2 when 
the outcome is continuous, 
and 


a 0 j = the site-level random intercept term, assumed distributed normal with 

mean = 0, and variance = T 2 . 

This model also assumes that the £.. are independent of a 0j . 

The estimate / 10 is the estimated average ITT impact of SEEMA. If / 10 is statistically significant and 
positive, we will reject the null hypothesis, and conclude that across all sites, the opportunity to 
participate in SEEMA has a positive impact on students/parents. 

To test whether the impacts are different for students with no prior participation, we will fit an 
interaction model of the form below (the two-level model is shown as a single, combined model). In 
this model Prior=l of student had prior SEEMA participation, and =0 if no prior SEEMA participation. 
If the interaction test is significant (i.e., test of whether / 30 =0) indicating different impacts for 
students with no prior participation, relative to students with prior participation, we will test 
whether there are impacts in each of the two subgroups. The impact estimate for students with no 
prior participation is / 10 , while the impact estimate for students with prior SEEMA participation is 

ho +ho ■ 


n 

Y ij = (r 00 + a oj) + no( trt ij) + r 2o(Yr ior y ) + noitrtij * Pr iorjj) + y^iprey - pre) + ^r k 0 (co\ ary -covar) + Sy 

k=5 
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Tests for nonresponse bias 


Our analytic sample for the impact analyses included only individuals who responded to both the 
baseline and follow-up surveys. Because the post-program response rate was lower than expected, 49 
we conducted several tests to identify potential nonresponse bias. Overall, there was no evidence 
that nonresponse introduced bias into the study. 

We compared the individuals who remained in the study with those who left the study at the time 
of follow-up data collection, in order to see if these groups differed along observable characteristics 
and baseline values on outcomes of interest. Specifically, in order to see if these groups differed, we 
(1) compared the observable characteristics of responders with nonresponders overall and in both 
the treatment and control groups; (2) compared the values on outcomes of interest at baseline of 
the responders and nonresponders; and (3) fit regression models to test the null hypotheses that 
there were no differences between the responders and nonresponders. The results of these tests 
are presented below. 

The overall response rate was 60 percent, and was the same among families assigned to both the 
treatment and control conditions (see Exhibit A. 2). Of the initial sample, 246 had previously 
participated in SEMAA. Individuals in the treatment group were equally likely to respond to the 
follow-up survey whether they had previously participated in SEMAA or not. In the control group, 66 
percent of the individuals who had previously participated responded, which was slightly higher 
than the 58 percent response rate among individuals who were new to SEMAA. 


Exhibit A.2: Overall Response Rate and by Prior Participation in SEMAA 




Full Sample 


Treatment Group 


Control Group 



Response 



Response 



Response 


Baseline 

Follow-up 

Rate 

Baseline 

Follow-up 

Rate 

Baseline Follow-up 

Rate 

Overall 

662 

397 

60.0% 

404 

241 

59.7% 

258 

156 

60.5% 

Prior SEMAA 

246 

152 

61 .8% 

163 

97 

59.5% 

83 

55 

66.3% 

Participation 
New to 
SEMAA 

392 

232 

59.2% 

228 

137 

60.1% 

164 

95 

57.9% 


The demographic characteristics of those who responded to the follow-up survey and those who did 
not are presented in Exhibit A. 3. 


We also conducted chi-square tests of independence to investigate whether the response rate 
varied by any background characteristics of students and families. Overall, response rates were 
similar across students and families with different demographic characteristics. However, response 
rates varied slightly across grades, with 7 th graders less likely to respond and 8 th graders more likely 


49 The target 75 percent response rate was set with a proposed incentive for participation; however, the 
Office of Management and Budget did not approve the use of incentives for this study, and the actual 
response rate for the study was 60 percent. 
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to respond than the overall sample average response rate. Students who had a family member who 
worked in a STEM field were also more likely to respond than their counterparts who did not have a 
family member in STEM. 


Exhibit A.3: Overall Response Rates and Demographic Characteristics of Responders and 
Nonresponders 




Response 

Rate 


Non- 

responders 



N 

Responders 

p-value 

Gender 





0.1881 

Male 

298 

62% 

43% 

48% 


Female 

364 

57 

57 

52 


Race 





0.22 

Latino 

49 

49% 

6% 

9% 


American Indiana 

2 

50 

0 

0 


Asian 

73 

66 

12 

10 


Pacific Islander 

2 

0 

0 

1 


Black 

421 

60 

66 

66 


White 

41 

66 

7 

6 


Multi 

21 

76 

2 

2 


No report 

43 

60 

7 

7 


Income 





0.6327 

Under 50,000 

265 

59% 

50% 

52% 


Over 50,0000 

256 

61 

50 

48 


School Grade 





0.0211 

Grade 4 

168 

61% 

26% 

24% 


Grade 5 

173 

61 

27 

25 


Grade 6 

142 

62 

22 

20 


Grade 7 

103 

46 

11 

21 


Grade 8 

76 

68 

13 

9 


Family member in STEM 





0.0305 

Yes 

433 

63% 

72% 

64% 


No 

229 

54 

33 

42 



We also looked for evidence of nonresponse bias in the outcomes of interest within the treatment 
and control groups. Exhibit A. 4 results of the tests that were conducted. There was no evidence of 
nonresponse bias, as there were no significant differences between responders and nonresponders 
within either the treatment of control groups. 
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Exhibit A.4: Results of Tests for Non-Bias 


> 

■ 

O 



Control 



Treatment 




Test for 
difference 



Test for 
difference 

Respondent 

Nonrespondent 

in means 

Respondent 

Nonrespondent 

in means 

Mean Std 

Mean Std 

p-value 

Mean Std 

Mean Std 

p-value 


Student 

Desire to do science 

Frequency of participation in informal 

science activity 

Number of informal science activities 
Interest in informal science activities 
Engaged in STEM activities 
Anxiety toward science 
Self-confidence in science 
Interest in high school STEM coursework 
Expectation of college attendance 
Interest in college STEM coursework 
Interest in STEM careers 
Value of science to society 
Parent 

Parent support for student's out-of-school 

science activities prev 3 

Parent support for student's out-of-school 

science activities next 3 

Parent encouragement of high school STEM 

coursework 

Parent perception of student's need to 
understand science/math as an adult 
Parent encouragement of STEM fields in 
college 


3.15 

0.46 

3.18 

0.55 

0.4567 

2.39 

1.15 

2.44 

1.28 

0.9855 

0.87 

0.85 

0.95 

0.92 

0.9664 

3.25 

0.57 

3.29 

0.52 

0.6241 

0.51 

0.31 

0.51 

0.28 

0.9533 

1.55 

0.45 

1.56 

0.50 

0.8751 

2.77 

0.29 

2.71 

0.25 

0.1117 

2.55 

0.97 

2.76 

0.94 

0.0742 

1.04 

0.23 

1.03 

0.17 

0.8390 

2.88 

0.82 

2.88 

0.89 

0.9148 

2.36 

0.81 

2.44 

0.89 

0.5290 

3.28 

0.40 

3.26 

0.47 

0.7936 


2.30 

1.14 

2.32 

1.15 

0.9498 

2.68 

1.05 

2.88 

1.04 

0.1574 

3.59 

0.45 

3.62 

0.44 

0.5116 

3.49 

0.59 

3.56 

0.63 

0.2843 

3.54 

0.44 

3.47 

0.51 

0.2404 


3.15 

0.55 

3.14 

0.54 

0.7583 

2.32 

1.19 

2.28 

1.17 

0.4411 

0.95 

0.85 

0.85 

0.85 

0.1662 

3.22 

0.63 

3.28 

0.64 

0.4556 

0.50 

0.28 

0.51 

0.29 

0.8728 

1.56 

0.52 

1.66 

0.54 

0.0993 

2.73 

0.31 

2.76 

0.27 

0.2236 

2.65 

0.99 

2.75 

1.00 

0.2846 

1.02 

0.15 

1.04 

0.20 

0.3561 

2.82 

0.90 

2.83 

0.95 

0.8561 

2.43 

0.78 

2.37 

0.87 

0.3705 

3.29 

0.47 

3.28 

0.46 

0.981 






2.08 

1.08 

2.23 

1.15 

0.3494 

2.55 

0.98 

2.66 

0.96 

0.3568 

3.60 

0.47 

3.62 

0.40 

0.6273 

3.59 

0.52 

3.55 

0.55 

0.6225 

3.51 

0.48 

3.54 

0.46 

0.9559 



Baseline Balance Testing 


Because the study randomly assigned families to treatment and control groups, we would expect 
the groups to be equivalent on baseline measures of the outcomes. However, because of the 
response rate, we conducted a series of tests to examine whether, among individuals in the final 
analytic sample, the treatment and control groups were equivalent in their baseline measures. We 
fit regression models to test for differences between the two groups at baseline on any outcome 
measures. Exhibit A. 5 presents the means and standard deviations, of the treatment and control 
group members on the outcomes included in the study, and the results of the statistical tests for 
differences between groups from the regression models. Across all outcomes included in the study 
the characteristics of the treatment and control groups were similar. 

Study Minimum Detectable Effects 

The minimum detectable standardized effect size (MDSES) is the smallest effect size that could have 
been detected by the study with 80 percent power. All of the MDSESs were of similar magnitude, 
ranging from 0.24 to 0.31, indicating that detecting a statistically significant difference on any of the 
outcomes would have required a treatment-control difference of roughly one-quarter to one-third 
of a standard deviation. By general standards in the educational evaluation literature, these would 
represent quite large effects. 50 Given the smaller than expected initial study sample, and the lower 
than expected response rate at follow-up, the resulting reduction in the final analytic sample size 
(n=396) led to lower statistical power, lower precision than is necessary to detect smaller effect 
sizes. 


Kane (2004, cited in Bloom, 2005) calculated that a year of elementary school attendance increases 
average test scores in reading and math by 0.25 standard deviations; the National Center for Education 
Statistics (1997, cited in Bloom, 2005) estimated a 0.26 standard deviation improvement in math scores 
from a year of high school. While useful as benchmarks, it should be noted that the standardized test 
scores used in these calculations are fundamentally different measures from the interest and attitude 
measures utilized in evaluating the impact of SEMAA. 
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Exhibit A.5:. Baseline Values on Outcomes Investigated 



Treatment 



Control 




Outcome 

Mean 

sd 

N 

Mean 

sd 

N 

t-test 

p-value 

Student 

Desire to do science 

3.15 

0.46 

239 

3.15 

0.55 

156 

-0.04 

0.9658 

Frequency of participation in informal science 
activities 

2.32 

1.19 

236 

2.39 

1.15 

255 

-0.6 

0.5483 

Number of informal science activities 

0.95 

0.85 

230 

0.87 

0.85 

148 

0.79 

0.4297 

Interest in informal science activities 

3.22 

0.63 

237 

3.25 

0.57 

155 

-0.42 

0.6774 

Engaged in STEM activities 

0.50 

0.28 

236 

0.51 

0.31 

155 

-0.2 

0.8387 

Anxiety toward science 

1.56 

0.52 

239 

1.55 

0.45 

156 

-0.05 

0.9608 

Self-confidence in science 

2.73 

0.31 

239 

2.77 

0.29 

156 

-1.23 

0.2204 

Interest in high school STEM coursework 

2.65 

0.99 

236 

2.55 

0.97 

152 

1.00 

0.3168 

Expectation of college attendance 

1.02 

0.15 

177 

1.04 

0.23 

109 

-0.62 

0.5328 

Interest in college STEM coursework 

2.82 

0.90 

234 

2.88 

0.82 

151 

-0.64 

0.5227 

Interest in STEM careers 

2.43 

0.78 

235 

2.36 

0.81 

153 

0.93 

0.355 

Value of science to society 

3.29 

0.47 

239 

3.28 

0.4 

156 

0.15 

0.8779 

Parent 

Parent support for student's out-of-school science 
prior activities 

2.08 

1.08 

239 

2.30 

1.14 

153 

-1.84 

0.0667 

Parent support for student's out-of-school science 
upcoming activities 

2.55 

0.98 

238 

2.68 

1.05 

153 

-1.14 

0.2561 

Parent encouragement of High School STEM 
coursework 

3.60 

0.47 

236 

3.59 

0.45 

152 

0.31 

0.7535 

Parent perception of student's need to understand 
science/math as an adult 

3.56 

0.52 

238 

3.49 

0.52 

154 

1.89 

0.0596 

Parent encouragement of STEM fields in college 

3.51 

0.48 

236 

3.54 

0.44 

151 

-0.39 

0.6998 


Appendix B: Instruments 
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Student surveys - pretest and posttest 
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SEMAA SITE DIRECTOR 
Protocol for Telephone Interview 

Site: Person Interviewed: 

Interviewer: Title: 

Interview Date/Time: Telephone#: 


I. Background Information 

A. Project Director 

1 . How long have you been project director of the SEMAA project at ? 

2. What are your main responsibilities as project director? 

3. Do you hold any other positions? 

B. Project Staffing 

4. Please list your staff and describe the role of each. 

(Note: If this information is provided in the SOP, confirm staff and roles.)* 

5. Who is responsible for the day-to-day operation of your project? 

C. Host Institution and Locale 

Note: Interviewers should obtain demographic information (from the Web or elsewhere) on 
the host institution, school districts, schools involved in the program as well as the 
community where program is located. 

6. I see that the host institution for your project is . 

What components of your program are housed at the host institution? Where?* 

7. What components of your program are housed elsewhere? 

(Probe: Are classes held at school sites? Which school districts and schools? 

Where are FCC activities held? Where is the AEL located?)* 

II. Administrative, Governance, and Decision-making Structure of Program 

8. Where in the administrative structure of the host institution is your project located? 

Why was the decision made to locate the project there? 

9. What is the reporting structure for the project within the host institution? 

(Probe: To whom do you report?) 

10. Does your project have an advisory board? What constituencies are represented on 
the board?* 

1 1 . What are the roles and responsibilities of the Advisory Board? How often does the 
Board meet?* After identify role, ask: Was the Board effective or helpful? In what 
ways? 


* Asterisk alerts the interviewer (I) that the information requested may be found in NASA reports. 

In these cases, I is instructed to record the information from the report and confirm it with the Page 1 of 5 

interviewee or record it and ask question. . .as discussed during the training prior to conducting interviews. 


Firicil 

III. Project Goals and History 

12. What would you say are your SEMAA program's main goals? 

(Probe: Have these goals changed over time?) 

13. How do these goals align with overall NASA goals for SEMAA?* 

14. How do these goals align with the host institution goals? 

15. Why did your institution wish to establish a SEMAA project? 

(Probe: Can you give me a brief history of what led to the establishment of your SEMAA 
project? Does this project build on previous institutional initiatives?) 

IV. Program Components and Functions 

A. Program Type 

16. What is your main program type? 

(Choose one: Saturday program, in-school program, or after-school program)* 

17. Has your main program type changed since you began your project? 

(Probe: If so, how? Why did you decide to adopt another program type? When? 

In your opinion, what program type do you think has the most impact?) 

B. Program Components and Activities 

18. Does your SEMAA project have any of the following core components?* 

NASA STEM Hands-On curriculum 

Aerospace Education Laboratory (AEL) 

Family Cafe 

19. Which activities do you offer under each of the above components? 

Curriculum 

a. In addition to the SEMAA curriculum, what additional experiences are provided to 
meet local needs? (i.e., career exploration and counseling and enrichment 
activities — field trips, guest speakers, mentoring, peer support groups, etc.) 

b. Please describe how the curriculum used in your program aligns with national and 
state standards. 

c. How are parents involved in instructional activities? How are private sector 
businesses and industries involved? 

AEL 

a. What activities are offered at your AEL as part of core program activities? 

(Probe: What simulations are offered?)* 

b. What special outreach programs does your project sponsor at the AEL? 

(Probe: What is the target audience? About how many participants do you usually 
attract per year? Per session?)* 

c. Do you offer other outreach activities that do not involve the AEL? 

(Probe: If so, please describe the activities, target audience and number attracted.) 


* Asterisk alerts the interviewer (I) that the information requested may be found in NASA reports. 

In these cases, I is instructed to record the information from the report and confirm it with the Page 2 of 5 

interviewee or record it and ask question. . .as discussed during the training prior to conducting interviews. 


Family Cafe 

What are typical Family Cafe activities?* 

(Probe: any activities specific to science or technology) 

Note: Grantees report regularly on frequency and length of sessions, numbers of 
participants, etc. Confirm these data during interview. 

20. What scientific topic areas does your project cover? (mark all that apply)* 

(Probe: does this vary by grade? Do you offer all topics in all grades?) 

(Note: prior to beginning data collection, consider alternate categories) 

a. Earth Science 

b. Areospace technology 

c. Human exploration and development of space 

d. Space science 

e. Other (specify) 

21 . a. Do you offer a summer session? 

(Probe: If so, please describe — length, where housed, content)* 

b. What percentage of all participants during the regular sessions (generally 3 sessions) 
typically attend your summer session? 


C. Student Recruitment. Selection and Retention 

22. Please describe the student population targeted for recruitment? 

(Probe: confirm grade levels, get demographic and achievement profile)* 

23. How are students recruited for your project? What recruitment strategies have you found 
to be most successful?* 

24. Does the project have an application/selection process? If so, please describe this 
process. 

25. On average, what percentage of students in your project continue to the following year? 
(Probe: does it vary by grade level? Is their greater attrition in the upper grades? 

Is there less re-enrollment in the upper grades?) 

26. How do you try to ensure that students continue in your project? 

27. How is student progress monitored? 

(Probe: Who is monitored? How often? By whom? What kind of information is used to 
monitor student progress?) 

28. Interviewer: skip this question if don't monitor student progress (q27). 

Do students receive systematic feedback on their progress? 

(Probe: How often? through what mechanismjs]?) 


* Asterisk alerts the interviewer (I) that the information requested may be found in NASA reports. 

In these cases, I is instructed to record the information from the report and confirm it with the Page 3 of 5 

interviewee or record it and ask question. . .as discussed during the training prior to conducting interviews. 


Pinsl 

D. Teacher Recruitment. Selection and Training 

29. How do you select teachers for your program? 

(Probe: What are selection criteria? Selection process? Do you hire both preservice and 
inservice teachers?)* 

30. Please describe the training that teachers in your project receive. 

(Probe: How long does the training last? Who provides it? What is covered by the training 
[content, pedagogy]?)* 

31 . On average, how long do teachers stay in your project? Why do they leave? 

(Probe: interviewer find out if leave the school or the project? Of those who leave, which 
share do so because they leave the school?) 

E. Parent Recruitment and Retention 


32. How are parents recruited into the Family Cafe component of the project?* 

33. What steps are taken by the project to ensure continued family participation?* 

34. On average, what percentage of parents participate in the Family Cafe? And what 
percentage continue participation from year to year? 

F. Collaborations/Partnerships and Leveraging of Resources 

35. Have you established formal partnerships or collaborative relationships with the following 
entities? If so, please identify the partner within each category, list the relationship 
(internal/external or financial/other partner) and give monetary value of contributions:* 

Partners: Name ^Internal F=Financial Value 

(circle! E=External Q=Other (£1 

a. school systems/districts? 

b. universities? 

c. research centers/programs? 

d. laboratories? 

e. industry/business entities? 

f. community groups? 

g. other? 

G. Sustainability 

36. Please describe your plan for sustaining the project beyond the funding period.* 

H. Evaluation and Reporting 

37. Are you conducting an evaluation of your project? If so, please describe it. 

38. Does your project use the data that you are required to report to NASA for planning or 
other purposes? If so, please describe how these data are used. 

39. Does your project track participants? If so, can you describe the process?* 

V. Project Outcomes 

* Asterisk alerts the interviewer (I) that the information requested may be found in NASA reports. 

In these cases, I is instructed to record the information from the report and confirm it with the Page 4 of 5 

interviewee or record it and ask question. . .as discussed during the training prior to conducting interviews. 


40. What do you see as your project’s main outcomes? 

41 . What percentage of participants in your project do you estimate have majored in a STEM 
degree in college? What percentage may be working in a STEM field? 

VI. Lessons Learned 

42. What have been the main lessons that you have learned in implementing your project? 

43. What have been the main factors that have helped you implement your project successfully? 

44. What have been your biggest challenges in implementing your project? 


* Asterisk alerts the interviewer (I) that the information requested may be found in NASA reports. 

In these cases, I is instructed to record the information from the report and confirm it with the Page 5 of 5 

interviewee or record it and ask question. . .as discussed during the training prior to conducting interviews. 
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Evaluation of NASA’s SEMAA (Science, Engineering, Mathematics, 
and Aerospace Academy) Program 

Student Survey 


1. Write your name neatly on this page. Do not write your name on any other pages. 

2. Read each question carefully. This is not a test: there are no right or wrong answers. We just 
want to know what you think. 

3. Use a pencil to mark your answers. If you want to change one of your answers, be sure to erase 
the old answer completely and mark the new answer darkly. 


1 - 10 / 

11-13/ 

14-38/ 

39-78/ 


79-80/ 

81-82/ 

83-86/ 


Please do not write your name on any other pages in this booklet. 


Please print your lirst and last name here on this page. Also tell us when you were born. Do not 
write your name on any other pages. 
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Sample Questions 

Some questions in the booklet ask you to mark just one answer. These questions say, “Check only 
one box.” Here is an example: 

How much do you like to do each of these activities? For each of these activities, 

check only one box: 


I really do I do not I really like 

not like it. like it. I like it. it. 



In question 1, you may check only one box to tell us how much you like to learn about plants. In 
question 2, you may check only one box to tell us how much you like to watch a movie about the 
ocean. 


For other questions in this booklet, you must check YES or NO for each item. Here is an example: 


3. Would you like to learn more about these topics? 


Yes No 

□ a 
a □ 

□ ^ 
A □ 
A □ 


How people in ancient times lived 
How clouds are formed 
What happened to the dinosaurs 
How to make a robot 

Types of animals that live in the rainforest 


If you do not understand these sample questions, please ask your parent to help you. 


Please go to the next page. 
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A. Background Information 

A1 . Why do you want to be in the SEMAA program this year? 


87/ 
88 / 
89/ 
90/ 
91/ 
92/ 
93/ 
94/ 
95/ 
96/ 
97/ 
98/ 
99-148/ 

Please go to the next page. 


Yes 

No 





□ , 

□2 

My brother or sister was in SEMAA before. 




□ , 

□2 

My brother or sister is excited about SEMAA. 




□ i 

□2 

1 think it would be fun. 




□ , 

□2 

My parent wants me to go. 




□ i 

□2 

1 want to learn more about science. 




□ , 

□2 

Other reason: 


B. Interest in Science 


Each of the following statements is about the study of science. For each statement check one box to tell 
us if you Strongly Disagree, Disagree, Agree, or Strongly Agree with the statement. 



STRONGLY 

DISAGREE 

DISAGREE 

AGREE 

STRONGLY 

AGREE 






B2. Science is something that 1 enjoy very 
much. 

□, 

□2 

□3 

□4 






B4. Science is easy for me. 

□1 

□2 

□ 3 

□4 
















B8. 1 don’t think 1 could do advanced 

science. 

□1 

□2 

□3 

□4 






B10. Science is helpful in understanding 
today’s world. 

□, 

□2 

□3 

□4 






B12. No matter how hard 1 try, 1 cannot 
understand science. 

□, 

□2 

□3 

□4 






B1 4. 1 like to understand the scientific 
explanations for things. 

□, 

□2 

□3 

□4 


Please go to the next page. 


149/ 

150/ 

151/ 

152/ 

153/ 

154/ 

155/ 

156/ 

157/ 

158/ 

159/ 

160/ 

161/ 

162/ 
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Check one box to tell us if you Strongly Disagree, Disagree, Agree, or Strongly Agree with each statement. C. Out of School Ac tivities 

Below is a list of activities that you might have done outside of school over this past summer. 

Over this past summer, how often did you do each of these activities? Check only one box for each activity. 

164/ 

165/ 

166/ 

167/ 

168/ 

169/ 

170/ 

171/ 

172/ 

173/ 

174/ 

175/ 

176/ 


Summer Activities Outside of 
School 

Not at all 
over the 
summer 

One 

Time 

About 
once a 
month 

About 
twice a 
month 

About 
once a 
week 

More 
than 
once a 
week 

Cl. 

Go to a science museum or zoo 

□ , 

□2 

□3 

□4 

□5 

□« 

C 2 . 

Read about science in a book, 
magazine, or on a website 

□ , 

□2 

□3 

□4 

□5 

□< 

C 3 . 

Take something apart, put 
something together or build 
something new 

□ , 

□2 

□3 

□4 

□5 

□ « 

C 4 . 

Watch a science program on TV 

□ , 

□2 

□3 

□4 

□5 

□ « 

C 5 . 

Try to figure out how something 
works 

□ , 

□2 

□3 

□4 

□5 

□ 6 

C 6 . 

Do an experiment 

□ i 

□2 

□3 

□4 

□5 

□ 6 

C 7 . 

Learn how to use a new tool, a 
new piece of equipment, or a 
new scientific instrument 

□ , 

□2 

□3 

□4 

□5 


C 8 . 

Use a science kit or do a science 
project at home 

□ i 

□2 

□3 

□4 

□5 

□ „ 

C 9 . 

Design something using your 
own ideas 

□ , 

□2 

□3 

□4 

□5 

□ * 

CIO. 

Play a video game 

□ , 

□2 

□3 

□4 

□5 

□ „ 

Cl 1 . 

Watch a movie at home or in a 
theater 

□ , 

□2 

□3 

□4 

□5 

□ „ 

C 12 . 

Use a computer or technology to 
solve a problem 

□ r 

□2 

□3 

□4 

□5 

□ „ 


Please go to the next page. 



STRONGLY 

DISAGREE 

DISAGREE 

AGREE 

STRONGLY 

AGREE 

B 1 5 . 1 often think, “1 cannot do this,” when a 
science assignment seems hard. 

□i 

□2 

□3 

□4 

B 16 . When 1 can’t immediately solve a 
science problem, 1 stick with it until 1 
have the solution. 

□t 

□2 

□3 

□4 

B 1 7 . Science is of great importance to a 
country's development. 


□2 

□3 

□4 

B 1 8 . It is important to know science in order 
to get a good job. 

□t 

□2 

□3 

□4 

B 1 9 . 1 like the challenge of science 
assignments. 

□i 

□2 

□3 

□4 

B 20 . It makes me nervous to even think 
about doing science. 

□i 

□2 

□3 

□4 

B 21 . It scares me to have to take a science 
class. 


□2 

□3 

□4 

B 22 . It is important to me to understand the 
work 1 do in science class. 


□2 

□3 

□4 

B 23 . 1 have a good feeling about science. 


□2 

□3 

□4 

B 24 . Science is one of my favorite subjects. 


□2 

□3 

□4 

B 25 . 1 really want to learn about science. 


□2 

□3 

□4 

B 26 . 1 do not do very well in science. 

□t 

□2 

□3 

□4 

B 27 . Sometimes 1 would like to learn more 
about science than the teacher covers 
at school. 


□2 

□3 

□4 

B 28 . 1 think science is boring. 

□t 

□2 

□3 

□4 


Please go to the next page. 


177/ 

178/ 

179/ 

180/ 

181/ 

182/ 

183/ 

184/ 

185/ 

186/ 

187/ 

188/ 


5 


6 





OMB Control No. 2700-0136 


OMB Control No. 2700-0136 


C13. 


During this past summer, did you do any of the following science or mathematics 


activities? 



Yes 

No 



□ i 

□ 2 

Study with a science or mathematics tutor 

189/ 

□ i 

□ 2 

Tutor someone else in science or mathematics 

190/ 

□ , 

□ 2 

Participate in the NASA SEMAA summer program 

191/ 

□ i 

□ 2 

Participate in a different science or math program 

192/ 

□ i 

□ 2 

(What is the name of this program? 

) 193/ 


194-242/ 

□ i 

□ 2 

Participate in any other science or math activities, please describe: 

243/ 


244-293/ 


On the next page are some activities. Make a check mark to show if you would really not like, 
would not like, would like, or would really like to do each activity. If you are not sure, check the 
question mark. Here are some examples: 



1 would 
really not 
like it 

1 would 
not like it 

1 would 
like it 

1 would 
really 
like it 

I’m not 
sure 

Eat a raw onion: 

w 

© 

© 

© 

7 

Clean up your bedroom: 

® 

& 

© 

© 

7 

Fly a kite: 

® 

® 


© 

7 

Eat some ice cream: 

® 


© 

~W 

7 

Read a biography of Benjamin Franklin: 

® 

® 

© 


y? 


Please go to the next page. 
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For each activity, make one check mark to show if you would really not like it, would not like it, would 
like it, would really like it, or if you are not sure. 

D. Science and Math in School 


Activities Outside of School 

1 would 
really not 
like it 

1 would 
not like it 

1 would 
like it 

1 would 
really like 
it 

Not Sure 

Cl 4. Go to a science museum or zoo 

©, 

©, 

©, 

©. 

? s 

Cl 5. Read about science in a book, 
magazine, or on a website 

©, 

®, 

©, 

©. 

? 8 

C16. Take something apart, put 
something together or build 
something new 


§j 

©, 

®. 

7 8 

Cl 7. Watch a science program on TV 


®, 

©, 

©. 

7 8 

Cl 8. Try to figure out how something 
works 

©, 

©, 

©, 

©. 

7 8 

Cl 9. Do an experiment 

©, 

© 

©, 

©. 

7 8 

C20. Learn how to use a new tool, a 
new piece of equipment, or a new 
scientific instrument 

©, 

©| 

©, 

©. 

7 8 

C21 . Use a science kit or do a science 
project at home 

®, 

©, 

©, 

©. 

7 8 

C22. Design something using your own 
ideas 

©, 

©, 

©. 

©. 

? 8 

C23. Play a video game 

®, 

®, 

©. 

©. 

? 8 

C24. Watch a movie at home or in a 
theater 

©, 

©| 

©, 

©. 

? 8 

C25. Use a computer or technology to 
solve a problem 

©, 

©, 

©, 

©. 

? 8 


294/ 

295/ 

296/ 

297/ 

298/ 

299/ 

300/ 

301/ 

302/ 

303/ 

304/ 

305/ 


D1 . Do you participate in your school’s science or engineering club? 

Check only one box. 

Qi Yes 
Q 2 No 

□3 My school does not have a science club. 

D2. Would you enjoy being in a science or engineering club at your school next year? 

Check only one box: 

Qi Yes 
Q 2 No 

Qs I am not sure 

D3. How many times have you done a project for a science or engineering fair? 
Check only one box: 

Qi Never 
Q 2 1-2 times 
Q 3 3-4 times 
Q 4 5-10 times 
Q 5 More than 1 0 times 

D4. Would you enjoy doing a project for a science or engineering fair? 

Check only one box: 


□ l 

Yes 

□ 2 

No 

□b 

Not sure 


D5. How many times have you gone to a science or engineering camp? 

Check only one box. 



□t 

Never 

Please go to the next page. 

□2 

1-2 times 


□3 

3-4 times 


□4 

5 or more times 


306/ 


307/ 


308/ 


309/ 


310/ 


Please go to the next page. 
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E. Classes in High School 


F. College 


When students get to high school, they usually get to choose some of the classes they want to take. For FI . Do you want to go to college after you finish high school? 

each class, make one check mark to tell us whether you would definitely not take it, probably not take it, □, I want to go to college after high school, 

probably take it, or would definitely take it. If you do not know what one of the classes is, check the first |— I 

box for that row. If you are not sure if you would take it, check the last box for that row. ^ am n °t sure a bout college. 

□ 3 I do not want to go to college after high school. 


Pretend that you are deciding what classes to take in college. For each class make one check mark to 
tell us if you would definitely not take it, would probably not take it, would probably take it, or if you 
would definitely take the class. If you do not know what one of the classes is, check the first box for that 
row. If you are not sure if you would take it, check the last box for that row. 

311/ 

312/ 

313/ 

314/ 

315/ 

316/ 

317/ 

318/ 

319/ 

320/ 

321/ 

322/ 

323/ 


Please go to the next page. 


COLLEGE 

CLASSES 

1 don’t 
know 
what this 
is 

1 would 
definitely 
not take 
this 

in college 

1 would 
probably 
not take 
this in 
college 

1 would 
probably 
take this 
in college 

1 would 
definitely 
take this 
in college 

I’m not 
sure if 1 
would 
take this 
in college 

F2. Art, music, or 
acting 

□i 

□2 

□3 

□4 

□ 5 

□ 6 

F3. Astronomy 


□2 

□3 

□4 

□5 

□ 6 

F4. Biology 


□2 

□3 

□4 

□s 

□c 

F5. Business 

□, 

□2 

□3 

□4 

□5 

□ < 

F6. Chemistry 

□t 

□2 

□3 

□4 

□5 

□ 6 

F7. Computer science 

□, 

□2 

□3 

□4 

□5 

□ 6 

F8. Engineering 

□, 

□2 

□3 

□4 

□5 

a . 

F9. Law 


□ 2 

□3 

□4 

□# 

□ 6 

F10. Literature (books 
and writing) 

□ r 

□2 

□3 

□4 

□5 

□ ' 

F1 1 . Math 

□ i 

□2 

□ 3 

□4 

□5 

□ 6 

FI 2. Medicine 

□ , 

□2 

□3 

□4 

□s 

□ « 

FI 3. Physics 

□ i 

□2 

□3 

□4 

□« 

□ 6 

FI 4. Education 

□ i 

□2 

□3 

□4 

□s 

□ « 

F15. History 

□ i 

□2 

□3 

□4 

□5 

□ 6 


HIGH SCHOOL 
CLASSES 

1 don’t 
know 
what 
this is 

1 would 
definitely 
not take 
this in high 
school 

1 would 
probably 
not take 
this in high 
school 

1 would 
probably 
take this 
in high 
school 

1 would 
definitely 
take this 
in high 
school 

I’m not 
sure if 1 
would take 
this in 
high 
school 

El. 

Earth Science 

□, 

□2 

□3 

□4 

□s 

□ 6 

E2. 

Life Science 

□, 

□2 

□3 

□4 

□5 

□ 6 

E3. 

Physical Science 

□, 

□2 

□3 

□4 

□5 

□ 6 

E4. 

Biology 

□, 

□2 

□3 

□4 

□5 

□ 6 

E5. 

Chemistry 

□1 

□2 

□3 

□4 

□5 

□ 6 

E6. 

Physics 

□, 

□2 

□3 

□4 

□5 

□ 6 

E7. 

Anatomy and 
Physiology 

□, 

□2 

□3 

□4 

□5 

□ « 

E8. 

Computer Science 

a. 

□2 

□3 

□4 

□* 

□ 6 

E9. 

Algebra 

a. 

□2 

□3 

□4 

□» 

□ 6 

E10. 

Geometry 

□, 

□2 

□3 

□4 


□ 6 

Ell. 

Statistics 

□, 

□2 

□3 

□4 

□5 

□ « 

E12. 

Calculus 

□, 

□2 

□3 

□4 

□ 5 

□ 6 

E13. 

Astronomy 

a. 

□2 

□3 

□4 

□5 

□ 6 


Please go to the next page. 


324/ 


325/ 

326/ 

327/ 

328/ 

329/ 

330/ 

331/ 

332/ 

333/ 

334/ 

335/ 

336/ 

337/ 

338/ 
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G. Jobs 


Below is a list of different types of jobs you might do when you are older. How much would you like to 
do these jobs? Check only one answer for each job. 


JOBS 

1 don’t 
know what 
this job is 

1 would 
really not 
like this 
job 

1 would 
not like 
this job 

1 would 
like 

this job 

1 would 
really like 
this job 

I’m not 
sure 

G1 . Scientist 

□ t 

©, 

© 

~©T 

©. 

□< 

G2. Entertainer (Actor, 
singer, dancer) 

□ t 


©, 

© 

© 

□« 

G3. College Professor 

□ t 

© 

© 

© 4 

© 

□< 

G4. Engineer 

□ , 

© 

© 

© 

© 

□« 

G5. Firefighter 

□ t 

© 

© 

© 

© 

□« 

G6. Computer Scientist 

□ t 

© 

© 

© 

© 

□« 

G7. Math Teacher 

o, 

© 

© 

© 

© 

□« 

G8. Military Officer (Army, 
Navy, Air Force, 
Marines) 

□t 

© 

© 

© 

© 

□« 

G9. Writer 

□t 

© 

~©r 

~©r 

© 

□« 

G10. Science Teacher 

□t 

©2 

© 

© 

© 

□« 

G11. Doctor 

□t 

© 

© 

© 

© 

□« 

G12. Athlete 

□t 

©2 

© 

© 

© 

□« 

G13. Astronaut 

□t 

© 

© 

© 

© 

□« 

G14. Lawyer 


© 

© 

~®r 

© 

□« 

G15. Construction Worker 

□t 

© 

© 

© 

© 

□« 

G16. BusinessPerson 

□, 

© 

© 

© 

© 

□« 

G17. Police Officer 

□t 

© 

© 

© 

© 

□« 


339/ 

340/ 

341/ 

342/ 

343/ 

344/ 

345/ 

346/ 

347/ 

348/ 

349/ 

350/ 

351/ 

352/ 

353/ 

354/ 

355/ 


G18. 


Are any of your friends or family members scientists, engineers or mathematicians? 

You may check more than one box. 


□i Mother/Stepmother 
l-l 2 Father/Stepfather 
Q 3 Foster Parent 

356/ 

357/ 

358/ 

359/ 

360/ 

361/ 

362/ 

363/ 

364/ 


Q 4 Aunt/Uncle 

□ 

O 5 Brother/Sister 


Qe Friend’s Parent 



Grandparent 

Family Friend/Neighbor 

Cousin 


G1 9. Did your parent or another adult help you do this survey? Check the answer that best 
describes how much help you got with this survey. Check only one box. 

Qi Yes, they helped me do almost all of the survey. 

□2 Yes, they helped me do some of the survey. 

□3 Yes, they helped me do 1 or 2 things on the survey. 

Q 4 No, I did all of the survey on my own. 

365/ 


Thank you for completing the survey ! 


Put your survey in the envelope labeled “Student Survey ” 
and seal it closed. 

Then give the en velope to your paren t. 
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Please go to the next page. 
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Evaluation of NASA’s SEMAA (Science, Engineering, Mathematics, 
and Aerospace Academy) Program 

1 - 10 / 

11-13/ 

Student Survey 

Follow-Up 

Whether or not you participated in the Fall 2009 SEMAA session, 
please complete this survey! 

1. Write your name neatly on this page. Do not write your name on any other pages. 

2. Read each question carefully. This is not a test: there are no right or wrong answers. We just 
want to know what you think. 

3. Use a pencil to mark your answers. If you want to change one of your answers, be sure to erase 
the old answer completely and mark the new answer darkly. 


Please print your first and last name here on this page. Also tell us when you were born. Do not 
write your name on any other pages. 



14-148/B 


Please do not write your name on any other pages in this booklet. 
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Sample Questions 

Some questions in the booklet ask you to mark just one answer. These questions say, “Check only 
one box.” Here is an example: 

How much do you like to do each of these activities? For each of these activities, 

check only one box: 


I really do I do not I really like 

not like it. like it. I like it. it. 



In question 1, you may check only one box to tell us how much you like to learn about plants. In 
question 2, you may check only one box to tell us how much you like to watch a movie about the 
ocean. 


For other questions in this booklet, you must check YES or NO for each item. Here is an example: 


3. Would you like to learn more about these topics? 


Yes No 

□ a 
a □ 

□ ^ 
A □ 
A □ 


How people in ancient times lived 
How clouds are formed 
What happened to the dinosaurs 
How to make a robot 

Types of animals that live in the rainforest 


If you do not understand these sample questions, please ask your parent to help you. 


Please go to the next page. 
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A. Interest in Science Check one box to tell us if you Strongly Disagree, Disagree, Agree, or Strongly Agree with each statement. 

Each of the following statements is about the study of science. For each statement check one box to tell 
us if you Strongly Disagree, Disagree, Agree, or Strongly Agree with the statement. 

149/ 

150/ 

151/ 

152/ 

153/ 

154/ 

155/ 

156/ 

157/ 

158/ 

159/ 

160/ 

161/ 

162/ 

Please go to the next page. 



STRONGLY 

DISAGREE 

DISAGREE 

AGREE 

STRONGLY 

AGREE 

A1 . Science is useful for solving problems 
of everyday life. 

□, 

□ 2 

□3 

□4 

A2. Science is something that 1 enjoy very 
much. 


□2 

□3 

□4 

A3. 1 like to read about science even when 

my teacher doesn’t ask me to. 


□2 

□3 

□4 

A4. Science is easy for me. 


□2 

□3 

□4 

A5. When 1 hear the word “science,” 1 have 
a feeling of dislike. 

□i 

□2 

□3 

□4 

A6. Most people should study some 
science. 


□2 

□3 

□4 

A7. Sometimes 1 read ahead in our science 
book at school. 


□2 

□3 

□4 

A8. 1 don’t think 1 could do advanced 

science. 


□2 

□3 

□4 

A9. 1 think 1 could do more difficult science 

work than what 1 do now. 


CM 

n 

□3 

□4 

A10. Science is helpful in understanding 
today’s world. 


□2 

□3 

□4 

All. 1 usually understand what we are talking 
about in science lessons at school. 


□2 

□3 

□4 

A12. No matter how hard 1 try, 1 cannot 
understand science. 


□2 

□3 

□4 

A1 3. 1 feel tense when someone talks to me 
about science. 

□i 

□2 

□3 

□4 

A14. 1 like to understand the scientific 
explanations for things. 


□2 

□3 

□4 



STRONGLY 

DISAGREE 

DISAGREE 

AGREE 

STRONGLY 

AGREE 

A1 5. 1 often think, “1 cannot do this,” when a 
science assignment seems hard. 

□, 

□2 

□3 

□4 

A16. When 1 can’t immediately solve a 
science problem, 1 stick with it until 1 
have the solution. 


□2 

□3 

□4 

A1 7. Science is of great importance to a 
country's development. 

□, 

□2 

□3 

□4 

A1 8. It is important to know science in order 
to get a good job. 

□, 

□2 

□3 

□4 

A1 9. 1 like the challenge of science 
assignments. 


□2 

□3 

□4 

A20. It makes me nervous to even think 
about doing science. 

□, 

□2 

□3 

□4 

A21 . It scares me to have to take a science 
class. 

□, 

□2 

□3 

□4 

A22. It is important to me to understand the 
work 1 do in science class. 

□, 

□2 

□3 

□4 

A23. 1 have a good feeling about science. 

□i 

□2 

□3 

□4 

A24. Science is one of my favorite subjects. 

□i 

□2 

□3 

□4 

A25. 1 really want to learn about science. 


□2 

□3 

□4 

A26. 1 do not do very well in science. 

□, 

□2 

□3 

□4 

A27. Sometimes 1 would like to learn more 
about science than the teacher covers 
at school. 

□, 

□2 

□3 

□4 

A28. 1 think science is boring. 

□, 

□2 

□3 

□4 


Please go to the next page. 


163/ 

164/ 

165/ 

166/ 

167/ 

168/ 

169/ 

170/ 

171/ 

172/ 

173/ 

174/ 

175/ 

176/ 
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B. Out of school activities 


Below is a list of activities that you might have done outside of school in the last three months. 


During the last three months, how often did you do each of these activities? Check only one box for 
each activity. 


Activities Outside of School 

Not at all 
during 
the last 3 
months 

One 

Time 

About 
once a 
month 

About 
twice a 
month 

About 
once a 
week 

More 
than 
once a 
week 

177/ 

178/ 

179/ 

180/ 

181/ 

182/ 

183/ 

184/ 

185/ 

186/ 

187/ 

188/ 

B1 . Go to a science museum or zoo 

□ , 

□ 2 

□ 3 

□4 

□ 5 

□a 

B2. Read about science in a book, 

magazine, or on a website 

□ i 

□2 

□3 

□4 

□5 

□ e 

B3. Take something apart, put 

something together or build 
something new 

□ t 

□2 

□3 

□4 

□ 5 

□a 

B4. Watch a science program on TV 

□ i 

□2 

□3 

□4 

□5 

□a 

B5. T ry to figure out how something 

works 

□ , 

□2 

□3 

□4 

□5 

CD 

□ 

B6. Do an experiment 

□ i 

□2 

□3 

□4 

□5 

□a 

B7. Learn how to use a new tool, a 

new piece of equipment, or a new 
scientific instrument 

□ t 

□2 

□3 

□4 

□ 5 

CD 

n 

B8. Use a science kit or do a science 

project at home 

□ , 

□2 

□3 

□4 

□5 

□a 

B9. Design something using your own 

ideas 

□ t 

□2 

□3 

□4 

□ 5 

CD 

□ 

BIO. Play a video game 

□ i 

□2 

□3 

□4 

□5 

□a 

B1 1 . Watch a movie at home or in a 
theater 

□ , 

□2 

□3 

□4 

□5 

□a 

B12. Use a computer or technology to 
solve a problem 

□ , 

□2 

□3 

□4 

□5 

□a 


B13. During the past three months, did you do any of the following science or mathematics 


activities? 



Yes 

No 



□ 1 

□ 2 

Study with a science or mathematics tutor 

189/ 

□ 1 

□ 2 

Tutor someone else in science or mathematics 

190/ 

□ 1 

□ 2 

Participate in the NASA SEMAA program on Saturdays 

191/ 

□ 1 

□ 2 

Participate in a different science or math program outside of school 

192/ 



(What is the name of this program? 

j 193-242/ 

□ 1 

□ 2 

Participate in any other science or math activities. Please describe: 

243 

244-293/B 

366-515/ 


Please go to the next page. 
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For each activity, make one check mark to show if you would really not like it, would not like it, would 
like it, would really like it, or if you are not sure. 


On the next page are some activities. Make a check mark to show if you would really not like, 
would not like, would like, or would really like to do each activity. If you are not sure, check the 
question mark. Here are some examples: 



1 would 
really not 
like it 

1 would 
not like it 

1 would 
like it 

1 would 
really 
like it 

I’m not 
sure 

Eat a raw onion: 

w 

© 

© 

© 

? 

Clean up your bedroom: 

® 

& 

© 

© 


Fly a kite: 

® 

® 

© 

© 

7 

Eat some ice cream: 

® 


© 

~w 

7 

Read a biography of Benjamin Franklin: 

® 

® 

© 

© 

y 


Please go to the next page. 


Activities Outside of School 

1 would 
really not 
like it 

1 would 
not like it 

1 would 
like it 

1 would 
really like 
it 

Not Sure 

B14. Go to a science museum or zoo 

®, 

© 

©, 

©. 

? 8 

B1 5. Read about science in a book, 
magazine, or on a website 

©, 

© 

©, 

©. 

? 8 

B16. Take something apart, put 
something together or build 
something new 

® 

©, 

©, 

© 

? 8 

B1 7. Watch a science program on TV 

©, 

© 

©, 

© 

? 8 

B18. Try to figure out how something 
works 

® 

© 

©, 

© 

? 8 

B19. Do an experiment 

©, 

© 

©, 

© 

? 8 

B20. Learn how to use a new tool, a 
new piece of equipment, or a new 
scientific instrument 

© 

©, 

@ 

CO 

© 

? 8 

B21 . Use a science kit or do a science 
project at home 

®, 

© 

©, 

© 

? 8 

B22. Design something using your own 
ideas 

© 

©, 

©, 

© 

? 8 

B23. Play a video game 

©, 

© 

©, 

© 

? 8 

B24. Watch a movie at home or in a 
theater 

© 

© 

CO 

© 

© 

? 8 

B25. Use a computer or technology to 
solve a problem 

©, 

© 

©, 

© 

? 8 


294 / 

295/ 

296/ 

297/ 

298/ 

299/ 

300/ 

301/ 

302/ 

303/ 

304/ 

305/ 


Please go to the next page. 
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Let’s compare how you felt during the summer about some activities and how you feel about these 
activities now. Do you enjoy these activities much less than, less than, about the same, more than, or 
much more than you did during the summer? For each activity, put a check in one box. 

516/ 
517/ 

518/ 

519/ 
520/ 
521/ 

522/ 

523/ 
524/ 
525/ 
526/ 
527/ 

Please go to the next page. 


1 enjoy ... 

Much less 
now than 
in the 
summer 

A little 
less now 
than in 
the 

summer 

About the 
same as 
in the 
summer 

A little 
more now 
than in the 
summer 

Much 
more now 
than in the 
summer 

B26. 

going to a science 
museum or zoo 

□1 

□2 

□3 

□4 

□5 

B27. 

reading about science in 
a book, magazine, or on a 
website 


□2 

□3 

□4 

□5 

B28. 

taking something apart, 
putting something 
together or building 
something new 


□2 

□3 

□4 

□ 5 

B29. 

watching a science 
program on TV 

□1 

□2 

□3 

□4 

□ 5 

B30. 

trying to figure out how 
something works 


□2 

□3 

□4 

□5 

B31 . 

doing an experiment 


□2 

□3 

□4 

□5 

B32. 

learning how to use a 
new tool, a new piece of 
equipment, or a new 
scientific instrument 

□1 

□2 

□3 

□4 

□5 

B33. 

using a science kit or 
doing a science project at 
home 


□2 

□3 

□4 

□5 

B34. 

designing something 
using my own ideas 

□1 

□2 

□3 

□4 

□5 

B35. 

playing a video game 

□1 

□2 

□3 

□4 

□5 

B36. 

watching a movie at 
home or in a theater 


□2 

□3 

□4 

□5 

B37. 

using computers or 
technology to solve 

□1 

□2 

□3 

□4 

□5 


problems 
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C. Science and Math in School 

Cl . Do you participate in your school’s science or engineering club? 

Check only one box. 

□ Yes 306/ 

1^2 No 

□ My school does not have a science club. 

C2. Would you enjoy being in a science or engineering club at your school next year? 

Check only one box: 

□ Yes 307/ 

Q 2 No 

Us I am not sure 

C3. How many times have you done a project for a science or engineering fair? 

Check only one box: 

□ 1 Never 308/ 

Q 2 1-2 times 

Q 3 3-4 times 
5-10 times 

Q 5 More than 10 times 

C4. Would you enjoy doing a project for a science or engineering fair? 

Check only one box: 

□ 1 Yes 309 / 

U 2 No 

Us Not sure 

C5. How many times have you gone to a science or engineering camp? 

Check only one box. 

□ 1 Never 310 / 

CL 1-2 times 

Q 3 3-4 times 
LI4 5 or more times 


Please go to the next page. 
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D. Classes in High School 

When students get to high school, they usually get to choose some of the classes they want to take. For 
each class, make one check mark to tell us whether you would definitely not take it, probably not take it, 
probably take it, or would definitely take it. If you do not know what one of the classes is, check the first 
box for that row. If you are not sure if you would take it, check the last box for that row. 


Pretend that you are deciding what classes to take in college. For each class make one check mark to 
tell us if you would definitely not take it, would probably not take it, would probably take it, or if you 
would definitely take the class. If you do not know what one of the classes is, check the first box for that 
row. If you are not sure if you would take it, check the last box for that row. 

311/ 

312/ 

313/ 

314/ 

315/ 

316/ 

317/ 

318/ 

319/ 

320/ 

321/ 

322/ 

323/ 


Please go to the next page. 


COLLEGE 

CLASSES 

1 don’t 
know 
what this 
is 

1 would 
definitely 
not take 
this 

in college 

1 would 
probably 
not take 
this in 
college 

1 would 
probably 
take this 
in college 

1 would 
definitely 
take this 
in college 

I’m not 
sure if 1 
would 
take this 
in college 

E2. 

Art, music, or 
acting 

□t 

□2 

□3 

□4 

□5 

□e 

E3. 

Astronomy 

□1 

□2 

□3 

□4 

□5 

□e 

E4. 

Biology 


□2 

□3 

□4 

□5 

□e 

E5. 

Business 


□2 

□3 

□4 

□5 

□e 

E6. 

Chemistry 

□1 

□2 

□3 

□4 

□5 

□e 

E7. 

Computer 

science 

□1 

□2 

□3 

□4 

□ 5 


E8. 

Engineering 

□1 

□2 

□3 

□4 

□5 

□e 

E9. 

Law 

□1 

□2 

□3 

□4 

□s 

□e 

E10. 

Literature (books 
and writing) 

□1 

□2 

□3 

□4 

□5 

CO 

n 

Ell. 

Math 

□1 

□2 

□3 

□4 

□5 


E12. 

Medicine 

□1 

□2 

□3 

□4 

□5 

□e 

E 13 . 

Physics 

□1 

□2 

□3 

□4 

□5 

□« 

E14. 

Education 


□2 

□3 

□4 

□5 


E15. 

History 


□2 

□3 

□4 

□5 

□s 


HIGH SCHOOL 
CLASSES 

1 don’t 
know 
what 
this is 

1 would 
definitely 
not take 
this in high 
school 

1 would 
probably 
not take 
this in high 
school 

1 would 
probably 
take this 
in high 
school 

1 would 
definitely 
take this 
in high 
school 

I’m not 
sure if 1 
would take 
this in 
high 
school 

D1. 

Earth Science 

□1 

□2 

□3 

□4 

□5 


D2. 

Life Science 


□2 

□3 

□4 

□5 


D3. 

Physical Science 

□1 

□2 

□3 

□4 

□5 


D4. 

Biology 

□1 

□2 

□3 

□4 

□ 5 

□e 

D5. 

Chemistry 

□1 

□2 

□3 

□4 

□5 

□« 

D6. 

Physics 

□1 

□2 

□3 

□4 

□ 5 


D7. 

Anatomy and 
Physiology 

□1 

□2 

□3 

□4 

□5 

□e 

D8. 

Computer Science 

□1 

□2 

□3 

□4 

□ 5 

□e 

D9. 

Algebra 

□1 

□2 

□3 

□4 

□ 5 


D10. Geometry 

□1 

□2 

□3 

□4 

□ 5 

□« 

Dll. 

Statistics 

□1 

□2 

□3 

□4 

□ 5 


D12. 

Calculus 

□1 

□2 

□3 

□4 

□5 

□e 

D13. 

Astronomy 

□1 

□2 

□3 

□4 

□5 

□e 


E. College 

324/ 

El . Do you want to go to college after you finish high school? 

Qi I want to go to college after high school. 

□2 I am not sure about college. 

□3 I do not want to go to college after high school. 


Please go to the next page. 


325/ 

326/ 

327/ 

328/ 

329/ 

330/ 

331/ 

332/ 

333/ 

334/ 

335/ 

336/ 

337/ 

338 / 
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F. Jobs 

Below is a list of different types of jobs you might do when you are older. How much would you like to 
do these jobs? Check only one answer for each job. . 


JOBS 

1 don’t 
know 
what this 
job is 

1 would 
really not 
like this 
job 

1 would 
not like 
this job 

1 would 
like 

this job 

1 would 
really like 
this job 

I’m not 
sure 

FI. 

Scientist 

□t 

® 

© 

« 

©. 

© 

□s 

F2. 

Entertainer (Actor, singer, 
dancer) 


© , 

© 

© 

© 

□. 

F3. 

College Professor 

□t 

© , 

© 

© 

© 

CD 

□ 

F4. 

Engineer 


© 

© 

© 

© 

□. 

F5. 

Firefighter 

□t 

© 

© 

© 

© 

□ a 

F6. 

Computer Scientist 


© 

© 

CO 

© 

© 

□ a 

F7. 

Math Teacher 

□t 

© 

© 

© 

© 

□ a 

F8. 

Military Officer (Army, Navy, 
Air Force, Marines) 

□t 

© 

© 

© 

© 

□ a 

F9. 

Writer 

□t 

© 

© 

© 

© 

□ a 

F10. 

Science Teacher 


© 

© 

CO 

© 

© 

□ a 

F1 1 . 

Doctor 


© 

© 

© 

© 

□ a 

FI 2. 

Athlete 


© 

© 

© 

© 

CO 

□ 

F 13 . 

Astronaut 

□t 

© 

© 

© 

© 

CD 

□ 

FI 4. 

Lawyer 

□t 

© 

© 

CO 

© 

© 

□ a 

F 15 . 

Construction Worker 


© 

© 

© 

© 

□ a 

FI 6. 

Business Person 


© 

© 

© 

© 

□ a 

FI 7. 

Police Officer 

□t 

© 

© 

CO 

© 

© 

□ a 


Please go to the next page. 


339/ 

340/ 

341/ 

342/ 

343/ 

344/ 

345/ 

346/ 

347/ 

348/ 

349/ 

350/ 

351/ 

352/ 

353/ 

354/ 

355/ 


FI 8. Did your parent or another adult help you do this survey? Check the answer that best 
describes how much help you got with this survey. Check only one box. 

Qi Yes, they helped me do almost all of the survey. 

□2 Yes, they helped me do some of the survey. 

□3 Yes, they helped me do 1 or 2 things on the survey. 

Q 4 No, I did all of the survey on my own. 


Thank you for completing the survey ! 


Put your survey in the envelope labeled “Student Survey ” 
and seal it closed. 

Then give the en velope to your parent. 


365/ 
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Evaluation of NASA’s SEMAA (Science, Engineering, Mathematics, 
and Aerospace Academy) Program 

1 - 10 / 

11-13/ 


Parent Survey 


1. Read and sign the Parent Consent and Permission Form. 

2. Read the Child Assent Form with your child and answer your child’s questions. 

3. Complete the Parent Survey (this booklet). 

4. Ask your child to complete the Student Survey. If your child does not understand an item on the 
Student Survey, you may explain the question to your child. However, please try not to tell your 
child how to answer the questions on the Student Survey. 
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Parent and Child Information 

Print your first and last names here: 


Parent’s 
First Name: 

Parent’s 
Last Name: 




14 - 38 / 39 - 78 / 

79-103/ 
104-143/ 

144-145/ 

146-147/ 
148-151/ 

152-153/ 


154-178/ 

179-218/ 

219-220/ 

221 - 222 / 

223-226/ 

227-228/ 


229-253/ 
254-293/ 

294-295/ 
296-297/ 
298-301/ 

302-303/ 

Please go to the next page. 


Child 3. 

First Name: 

Last Name: 


Birthdate: (month) 

(date) (vear) 


What grade is this child entering in school this year? 


d Kindergarten 

d 4 th grade 

d 9 th grade 

d 1 st grade 

d 5 th grade 

d 10 th grade 

d 2 nd grade 

d 6 th grade 

d 11 th grade 

d 3 rd grade 

d 7 th grade 

d 12 th grade 


d 8 th grade 



Child 2. 

First Name: 

Last Name: 


Birthdate: (month) 

(date) (vear) 


What grade is this child entering in school this year? 


d Kindergarten 

d 4 th grade 

d g® 1 grade 

d 1 st grade 

d 5 th grade 

d 10 th grade 

d 2 nd grade 

d 6 th grade 

d 1 1 th grade 

d 3 rd grade 

d 7 th grade 

d 12 th grade 


d 8 th grade 



Please tell us about each child who is applying to SEMAA this year. 


Child 1. 

First Name: 

Last Name: 


Birthdate: (month) 

(date) (vear) 


What grade is this child entering in school this year? 


d Kindergarten 

d 4 th grade 

d 9* grade 

d 1 st grade 

d 5 th grade 

d 10 ,fl grade 

d 2 nd grade 

d 6 th grade 

d 11 th grade 

d 3 ,d grade 

d 7 th grade 

d 12 th grade 


d 8 th grade 
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Do you have more than one child who is applying to SEMAA this year? 

□2 No Yes 

If two or more children applying to SEMAA will be in the 4 th - 8 th grade, pick 
just one child as follows: 

• If your last name begins with A-M, pick the oldest child . Write this 
child’s name in Box A below. 

• If your last name begins with N-Z, pick the second oldest child . Write 
this child’s name in Box A below. 



304/ 


• Write your child’s name in Box A. 

• When completing this Parent Survey, think about the child named in Box A. 

• Ask the child named in Box A to complete the Student Survey. 


Box A 

Child’s Child’s 

First Name: Last Name: 

This child should be in 4 th , 5 th , 6 th , 7 th , or 8 th grade in the 2009-2010 school year. This child 
should complete the Student Survey while you complete the Parent Survey. 


305-329/ 

330-369/ 


Please do not write your name or your child’s name on any other pages in this Parent Survey. 


Please go to the next page to complete the Parent Survey. 


A. Background Information 


Questions on this survey ask about the child whose name you have listed inside Box A on the 
preceding page. 


A1. What is your child’s gender? Qi Male Q 2 Female 


What is your relationship 

to this child? 

Mother/Stepmother 

06 Other male relative 

CJ2 Father/Stepfather 

O7 Female guardian 

O3 Grandfather 

Os Male guardian 

L-I4 Grandmother 

O9 Other: 

Os Other female relative 



370/ 
371 -372/ 


373-396/ 


A3. What is your child’s grade in school this year? 

□4 th a 5 th a6 th □7 th a 8 th 

397-398/ 

A4. How did you find out about the SEMAA program? Check all that apply. 


Oi Information sent home from school 

Q6 Newspaper advertisement 

399/ 

O2 Previous participation 

O7 Newspaper article 

400/ 

O3 Word of mouth 

Qs Web site 

401/ 

O4 Radio 

Q9 Brochure 

402/ 

O5 Other (dease SDecifvt: 


403/ 



404/ 

405/ 

406/ 

407/ 

408-457/ 


Please go 

to the next page. 
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The SEMAA program has two kinds of sessions. There are sessions held during the school year 
and there are summer sessions held when children are not in school. 

A7. What are your reasons for supporting your child’s application to the SEMAA program this 
year? You may check more than one box. 

458/ 

Qi My child attended SEMAA previously 

470/ 

A5. Has your child ever attended a SEMAA summer session? 
Qi Yes Q2 No Q 8 I’m not sure 

Q2 My child’s brother/sister attended SEMAA previously 

471/ 

Q3 My child is excited about SEMAA 

472/ 

If NO, please skip to Question A6. 

A5a. If YES, how many summer SEMAA sessions has your child participated in? 

Check only one box. 

459/ 

1— li 1 summer session 

Q4 My child’s brother/sister is excited about SEMAA 

473/ 

Q5 A friend of my child is excited about SEMAA 

474/ 

^6 1 am looking for a fun, hands-on program for my child 

475/ 

Q7 1 am looking for something educational for my child 

476/ 

Q2 2-4 summer sessions 

Qs 1 want my child to have something to do on the weekend 

477/ 

Q3 5-7 summer sessions 

Q9 My child does not like science or math 

478/ 

Q4 8 summer sessions 

Q10 My child enjoys science or math 

479-480/ 

460/ 

A6. Has vour child previously attended the SEMAA oroaram durina a reaular school year? 
Oi Yes O2 No 1^8 I’m not sure 

Q11 1 want my child to do better in school in general 

481-482/ 

Q12 1 want my child to do better in science/math in particular 

483-484/ 

If NO, please skip to Question A7. 

O13 My child is good at science/math 

485-486/ 

Q14 1 want to build my child’s confidence in science/math 

487-488/ 

A6a. If YES, how many SEMAA sessions durina the reaular school year, has vour 

1— 1 1 5 Other (please specify) : 

489-490/ 

child participated in? Please do not include participation in SEMAA summer 
sessions. Check only one box. 

461/ 


491-540/ 

Qi 1 school year SEMAA session before 


Please go to the next page. 


Q2 2-4 school year SEMAA sessions before 
Q3 5-7 school year SEMAA sessions before 
Q4 8 school year SEMAA sessions before 


A6b. If your child has ever participated in a school-year SEMAA session before this 
year, please check which grade or grades they were in when they participated in 
the past. You may check more than one box. 

□ Kindergarten □l st □2 nd □3 rd 04 th □5 th □6 th G/* 1 

462/ 463/ 464/ 465/ 
466/ 467/ 468/ 469/ 

Please go to the next page. 
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B. Student’s Interest in Science 


The following statements are about you, your child and science. Please check one box for each row to 
indicate whether you Strongly Disagree, Disagree, Agree, or Strongly Agree with each statement. 



Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 


B1 . Science is useful for solving problems of 
everyday life. 

□ i 

□ 2 

□3 

□4 

541 / 

B2. Science is something that my child enjoys 
very much. 

□ i 

□2 

□3 

□4 

542 / 

B3. My child likes to read about science even 
when the teacher doesn’t ask him/her to. 

□ i 

□2 

□3 

□4 

543 / 

B4. Science is easy for my child. 

□ i 

□2 

□3 

□4 

544 / 

B5. When my child hears the word “science,” 
he/she has a feeling of dislike. 

□ i 

□ 2 

□3 

□4 

545 / 

B6. Most people should study some science. 

□ i 

□2 

□3 

□4 

546 / 

B7. Sometimes my child reads ahead in the 
science book he/she uses in school. 

□ i 

□2 

□3 

□4 

547 / 

B8. 1 don’t think my child could do advanced 

science. 

□ i 

□2 

□3 

□4 

548 / 

B9. 1 think my child could do more difficult 

science work than what he/she does now. 

□ i 

□2 

□3 

□4 

549 / 

BIO. Science is helpful in understanding today’s 
world. 

□ i 

□2 

□3 

□4 

550 / 

B1 1 . My child usually understands science when it 
is talked about in class at school. 

□ i 

□2 

□3 

□4 

551 / 

B12. No matter how hard he/she tries, my child 
cannot understand science. 

□ i 

□2 

□3 

□4 

552 / 

B1 3. My child feels tense when someone talks to 
him/her about science. 

□ i 

□ 2 

□3 

□4 

553 / 

B14. My child likes to understand the scientific 
explanation for things. 

□ i 

□2 

□ 3 

□4 

554 / 


Check one box to tell us if you Strongly Disagree, Disagree, Agree, or Strongly Agree with each 
statement. 


Statements 

Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

B15. My child often thinks, “1 cannot do this,” 
when a science assignment seems hard. 

□ 1 

□2 

□3 

□4 

B16. When my child can’t immediately solve a 
science problem, he/she sticks with it until a 
solution is reached. 

□ 1 

□2 

□3 

□4 

B1 7. Science is of great importance to a country's 
development. 

□ 1 

□2 

□3 

□4 

B1 8. It is important to know science in order to 
get a good job. 

□ 1 

□2 

□3 

□4 

B1 9. My child likes the challenge of science 
assignments. 

□ 1 

□2 

□3 

□4 

B20. It makes my child nervous to even think 
about doing science. 

□ 1 

□2 

□3 

□4 

B21 . It scares my child to have to take a science 
class. 

□ 1 

□2 

□3 

□4 

B22. My child thinks it is important to understand 
the schoolwork he/she does in science. 

□ 1 

□2 

□3 

□4 

B23. My child has a good feeling about science. 

□ 1 

□2 

□3 

□4 

B24. Science is one of my child’s favorite 
subjects. 

□ 1 

□2 

□3 

□4 

B25. My child really wants to learn about science. 

□ 1 

□2 

□3 

□4 

B26. My child does not do very well in science. 

□ 1 

□2 

□3 

□4 

B27. Sometimes my child wants to learn more 
about science than the teacher covers at 
school. 

□ 1 

□2 

□3 

□4 

B28. My child thinks science is boring. 

□ 1 

□2 

□3 

□4 


Please go to the next page. 


Please go to the next page. 


555/ 

556 / 

557 / 

558 / 

559 / 

560 / 

561 / 

562 / 

563 / 

564 / 

565 / 

566 / 

567 / 

568 / 
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C. Out of School Activities 


Below is a list of activities that you and your child may or may not have shared over the summer. 


Over the summer, how often did you and your child together do this activity? Check only one box for each 
activity. 



Not at all 
over the 
summer 

One 

time 

About 
once a 
month 

About 

twice 

a 

month 

About 
once a 
week 

More 
than 
once a 
week 

569/ 

570/ 

571/ 

572/ 

573/ 

574/ 

575/ 

576/ 

577/ 

578/ 

579/ 

580/ 

Cl . Go to a science museum or zoo 

□ , 

□2 

□3 

□4 

□5 

□e 

C2. Read about science in a book, 
magazine, or website 

□ , 

□ 2 

□3 

□4 

□5 

□e 

C3. Have your child take something 
apart, put something together or 
build something new 

□ i 

□2 

□3 

□4 

□5 

□e 

C4. Watch a science program on TV 

□ , 

□ 2 

□3 

□4 

□5 

□s 

C5. T ry to figure out how something 
works 

□ , 

□2 

□3 

□4 

□5 

□e 

C6. Do an experiment 

□ , 

□ 2 

□3 

□4 

□5 

□ 6 

C7. Have your child learn how to use a 
new tool, a new piece of equipment, 
or a new scientific instrument 

□ l 

□2 

□3 

□4 

□5 

□e 

C8. Use a science kit or do a science 
project at home 

□ , 

□ 2 

□3 

□4 

□5 

□s 

C9. Design something using your child's 
ideas 

□ i 

□2 

□3 

□4 

□5 

□e 

CIO. Play a video game 

□ , 

□ 2 

□3 

□4 

□5 

□ 6 

Cl 1 . Watch a movie at home or in a 
theater 

□ , 

□2 

□3 

□4 

□5 

□e 

Cl 2. Use a computer or technology to 
solve a problem 

□ , 

□ 2 

□3 

□4 

□5 

□ 6 


Cl 3. During this summer, did your child participate in any of the following science or 
mathematics activities? 


Yes 

No 



□ , 

□ 2 

Study with a science or mathematics tutor 

581/ 

□ , 

□ 2 

Tutor someone else in science or mathematics 

582/ 

□ , 

□ 2 

Participate in the NASA SEMAA summer program 

583/ 

□ , 

□ 2 

Participate in a different science or math program 

584/ 



(What is the name of this program? 

) 

585-634/ 



(Who is the sponsor of this program? 

) 

635-684/ 

□ , 

□ 2 

Participate in any other science or mathematics activities 
Please describe: 

685/ 


686-735/ 


Please go to the next page. 


Please go to the next page. 
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In the next 3 months, how often do you think that you and your child together will do each of these 
activities? Check only one box for each activity. 



Not at all 
in the 
next 3 
months 

One 

time 

About 
once a 
month 

About 
twice a 
month 

About 
once a 
week 

More 
than 
once a 
week 

Cl 4. Go to a science museum or zoo 

□, 

□2 

□3 

□4 

□5 

□e 

Cl 5. Read about science in a book, 
magazine, or on a website 


□ 2 

□3 

□4 

□5 

□e 

Cl 6. Have your child take something 
apart, put something together or 
build something new 

□1 

□2 

□3 

□4 

□5 

□e 

Cl 7. Watch a science program on TV 

□1 

□2 

□3 

□ 4 

□5 

□ 6 

Cl 8. Try to figure out how something 
works 

□, 

□2 

□3 

□4 

□5 

□e 

Cl 9. Do an experiment 

a. 

□2 

□3 

□ 4 

□5 

□ 6 

C20. Have your child learn how to use a 
new tool, a new piece of equipment, 
or a new scientific instrument 

□, 

□2 

□3 

□4 

□5 

□e 

C21 . Use a science kit or do a science 
project at home 


□2 

□3 

□4 

□5 

□e 

C22. Design something using your child’s 
ideas 

□1 

□2 

□3 

□4 

□5 

□e 

C23. Play a video game 


□2 

□3 

□4 

□5 

□ 6 

C24. Watch a movie at home or in a 
theater 

□1 

□2 

□3 

□4 

□5 

□e 

C25. Use a computer or technology to 
solve a problem 


□2 

□3 

□4 

□5 

□e 


736/ 

737/ 

738/ 

739/ 

740/ 

741/ 

742/ 

743/ 

744/ 

745/ 

746/ 

747/ 


Please go to the next page. 
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D. Science and Math in School 

D1 . My child’s end-of-year grade in science class last year was (select the one box that 
best matches the grade your child received): 

748/ 

Qi “A” or Excellent 
Q 2 “B” or Good 
□3 “C” or Satisfactory 
□4 “D” or Barely Passing 
□5 “F” or Failing 
Q6 I don’t know 

□7 Not applicable, my child did not receive a science grade last year 
D2. I believe my child’s ability in science is... (Check only one box) 

749/ 

□ 1 Much higher than other children his/her age 
□2 Somewhat higher than other children his/her age 
□3 About the same as other children his/her age 
□4 Somewhat lower than other children his/her age 
□s Much lower than other children his/her age 

D3. My child’s end-of-year grade in math class last year was (select the one box that best 
matches the grade your child received): 

750/ 

Qi “A” or Excellent 
Q 2 “B” or Good 
□3 "C" or Satisfactory 
□4 “D” or Barely Passing 
□5 “F” or Failing 
^6 I don’t know 

□7 Not applicable, my child did not receive a science grade last year 
D4. I believe my child’s ability in math is... (Check only one box) 

751/ 

Qi Much higher than other children his/her age 
□2 Somewhat higher than other children his/her age 
□3 About the same as other children his/her age 
□4 Somewhat lower than other children his/her age 
□5 Much lower than other children his/her age 

Please go to the next page. 
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E. Educational and Career Expectations 

When your child reaches high school, there may be classes that you would encourage your child to take, 
even if the class is not required. 

How much would you encourage your child to take each of the following classes in high school? 
Please check only one box for each class. 


752/ 

753/ 

754/ 

755/ 

756/ 

757/ 

758/ 

759/ 

760/ 

761/ 

762/ 

763/ 

764/ 

765/ 

766/ 

767/ 

768/ 

769/ 


HIGH SCHOOL CLASSES 

Would 

definitely not 
encourage 

Would 
probably not 
encourage 

Would 

probably 

encourage 

Would 

definitely 

encourage 

El. Physics 

□i 

□ 2 

□3 

□4 

E2. Biology 


□2 

□3 

□4 

E3. Chemistry 

□i 

□2 

□3 

□4 

E4. Art 


□2 

□3 

□4 

E5. Earth Science 

□i 

□2 

□3 

□4 

E6. Astronomy 

□i 

□2 

□3 

□4 

E7. Computer science 


□2 

□3 

□4 

E8. Anatomy & Physiology 


□2 

□3 

□4 

E9. Music 

□i 

□2 

□3 

□4 

E10. Advanced Physics 


□2 

□3 

□4 

Ell. Advanced Biology 

□i 

□2 

□3 

□4 

El 2. Advanced Chemistry 


□2 

□3 

□4 

El 3. Algebra 


□2 

□3 

□4 

El 4. Geometry 


□2 

□3 

□4 

El 5. Trigonometry 

□i 

□2 

□3 

□4 

El 6. Calculus 


□2 

□3 

□4 

El 7. Statistics 


□2 

□3 

□4 

El 8. History 


□2 

□3 

□4 


Please check one box to tell us if you Strongly Disagree, Disagree, Agree, or Strongly Agree with each 
statement. 



Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

El 9. When my child is an adult, he/she will 
need a good understanding of basic 
science. 

□ , 

□2 

□3 

□4 

E20. When my child is an adult, he/she will 
need a good understanding of 

advanced science. 

□ , 

□ 2 

□3 

□ 4 

E21 . When my child is an adult, he/she will 
need a good understanding of basic 
mathematics. 

□ , 

□ 2 

□3 

□4 

E22. When my child is an adult, he/she will 
need a good understanding of 

advanced mathematics. 

□ , 

□ 2 

□3 

□ 4 


Please go to the next page. 


Please go to the next page. 
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If your child goes to college, he or she will have to choose a major field of study in which to earn a college 
degree. 

If your child were interested in many different fields, how much would you encourage your child to earn a 
degree in each of the following? Please check only one box for each type of degree. 


COLLEGE DEGREE 

Would 

definitely not 
encourage 

Would 
probably not 
encourage 

Would 

probably 

encourage 

Would 

definitely 

encourage 

E23. Physics 

□t 

□ 2 

□3 

□4 

E24. Biology 

□t 

□2 

□3 

□4 

E25. Computer science 

□i 

□2 

□3 

□4 

E26. Chemistry 


□2 

□3 

□4 

E27. Literature (books and writing) 


□2 

□3 

□4 

E28. History 


□2 

□3 

□4 

E29. Engineering 

□i 

□2 

□3 

□4 

E30. Mathematics 

□i 

□2 

□3 

□4 

E31 . Art, music, or acting 


□2 

□3 

□4 

E32. Medicine 

□i 

□2 

□3 

□4 

E33. Business, accounting, or 
finance 

□t 

□2 

□3 

□4 

E34. Education 

□i 

□2 

□3 

□4 

E35. Law 

□i 

□2 

□3 

□4 

E36. Astronomy 


□2 

□3 

□4 


Please go to the next page. 


Section F. About You and Your Child 



FI. 

What is your child’s ethnicity? 
Qi Hispanic or Latino 
Q 2 Not Hispanic or Latino 
Q 3 Choose not to answer 


788/ 

774/ 

F2. 

What is your child’s race? Mark one or more: 



775/ 


Qi American Indian or Alaska Native 


789/ 



Q 2 Asian 


790/ 

776/ 


Q 3 Native Hawaiian or other Pacific Islander 


791/ 



Q 4 Black or African American 


792/ 

777/ 


Q 5 White 


793/ 

778/ 


1^6 Choose not to answer 


794/ 

779/ 

F3. 

What language or languages do you and your child use at home? 

more than one box. 

You may check 


780/ 


Qi English ^6 Russian 


795/ 



Q 2 Spanish Q 7 Arabic 


796/ 

781/ 


Q 3 French Qs Chinese 


797/ 



O 4 Italian Q 9 Other, please specify: 

804-828/ 

798/ 

782/ 


Q 5 German 


799/ 





800/ 

783/ 




801/ 





802/ 





803/ 

784/ 





785/ 

F4. 

What language or languages does your child use at school? You may check more 



than one box. 



786/ 


Qi English ^6 Russian 


829/ 



Q 2 Spanish Q 7 Arabic 


830/ 

787/ 


Q 3 French Qs Chinese 


831/ 



O 4 Italian Q 9 Other, please specify: 

838-862/ 

832/ 



Q 5 German 


833/ 





834/ 





835/ 





836/ 





837/ 


Please go to the next page. 
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F5. What is the highest level of education that you have completed? What is the highest 
level of education that your child’s other parent or caregiver has completed? 

Child’s other 



You 

parent/caregiver 

Some High School 

□ 1 

□s 

High School (diploma or GED) 

□2 

□ 9 

Some College or University 

□3 

QlO 

Community college degree (AA) 

□4 

□ ll 

Bachelor’s Degree (BA or BS) 

□5 

Ol2 

Some Graduate School 

□e 

L_)l3 

Graduate Degree (Master’s, PhD, MD, JD, 
EdD) 

□7 

Ql4 


F6. Please indicate if you, or another adult your child knows well, are working in one of the 
following areas (check all that apply): 

Another adult my child 
I work in: knows well works in: 

866/ 867/ 
868/ 869/ 
870/ 871/ 
872/ 873/ 
874/ 875/ 
876/ 877/ 
878/ 879/ 
880/ 881/ 
882/ 883/ 

None of the Above □ 1 Q 2 884/ 885/ 



F7. Do you currently work for NASA? 
Qi Yes Q 2 No 


886 / 


Please go to the next page. 
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F8. Does your household currently have internet access? 887/ 

Qi Yes Q 2 No 

F9. About how often does your child access the internet from home (either with or without 

your supervision)? Check only one box. 888/ 

Qi Never 

Q 2 Less than once a week 
Q 3 About once a week 
2-3 times a week 
Q 5 4-6 times a week 

7 or more times a week 

F10. Please indicate, to the best of your knowledge, the annual household income for the 
household in which your child primarily lives. Check only one box. 

889/ 

□ 0 Under $10,000 

Qi $10,001 -$20,000 

□ 2 $20,001 -30,000 

□ 3 $30,001 -40,000 
□4 $40,001 -50,000 

□ 5 $50,001 -$60,000 
Qe Over $60,000 

O 7 Choose not to answer 


Thank you for completing the survey ! 


1. After completing the Parent Survey: 

Put it in the privacy envelope labeled “Parent Survey,” wet the seal, and close the flap. 


2. After your child has completed the Student Survey: 

Have your child place the completed survey in the envelope labeled “Student Survey,” wet the seal 
and close the flap. 


3. Place the following into the large pre-paid return envelope: 

• SEMAA application form 

• A signed copy of the Parent Consent and Permission Form 

• A signed copy of the Child Assent Form 

• The two privacy envelopes with completed surveys. 


4. Drop the return envelope at any US Post Office or in any Postal Drop Box. Thank you! 
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Evaluation of NASA’s SEMAA (Science, Engineering, Mathematics, 
and Aerospace Academy) Program 

1 - 10 / 

11-13/ 

Parent Survey 

Follow-Up 

Whether or not your child was enrolled in the Fall 2009 SEMAA 
session, please complete this survey and ask your child to complete 
the Student Survey! 

1. Complete the Parent Survey (this booklet). 

2. Ask your child to complete the Student Survey. If your child does not understand an item on 
the Student Survey, you may explain the question to your child. However, please try not to tell 
your child how to answer the questions on the Student Survey. 


Please give the Student Survey to the same child who completed the first student survey about two 
months ago. 


Are you the same individual who filled out the first Parent Survey when your child was 
applying to the SEMAA program? 

□ Yes n No 1040/ 


Please print your first and last name: 

Parent’s First Name: Last Name: 

Please print your child’s first and last name: 

Child’s First Name: Last Name: 

What month, date, and year was this child born? 

Month: Date: Year: 


14-370/B 


Please do not write your name or your child’s name on any other pages. 
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A. Background Information 


B. Student’s Interest in Science 


Questions on this survey ask about the child whose name you have listed inside the box on the 
preceding page. 


The following statements are about you, your child and science. Please check one box for each row to 
indicate whether you Strongly Disagree, Disagree, Agree, or Strongly Agree with each statement. 


A1 . What is your relationship to this child? 


di Mother/Stepmother 

CU Other male relative 

371-372/ 

O 2 Father/Stepfather 

O 7 Female guardian 

373-396/ 

O 3 Grandfather 

Male guardian 

397-540/B 

O 4 Grandmother 

Q 9 Other: 


O 5 Other female relative 




A2. Did your child participate in the Fall 2009 SEMAA session this year? 

□ i Yes — ► Go to A2a. □2 No — » SKIP ahead to the next page. 1041 / 

A2a. How many Family Cafe/Parent Cafe sessions did you attend? 


□ 

0 sessions 

d 3 sessions 

d 

6 sessions 

□ 

1 session 

n 4 sessions 

d 

7 sessions 

□ 

2 sessions 

d 5 sessions 

d 

8 sessions 


1042/ 1043/ 1044/ 1045/ 1046/ 
1047/ 1048/ 1049/ 1050/ 


A2b. How many Saturday sessions did your child attend this Fall? 


d 

0 sessions 

d 

3 sessions 

d 

6 sessions 

d 

1 session 

d 

4 sessions 

d 

7 sessions 

d 

2 sessions 

d 

5 sessions 

d 

8 sessions 


1051/ 1052/ 1053/ 1054/ 1055/ 
1056/ 1057/ 1058/ 1059/ 

Please go to the next page. 



Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

B1 . Science is useful for solving problems of 
everyday life. 

□, 

□2 

□3 

□4 

B2. Science is something that my child enjoys 
very much. 


□2 

□3 

□4 

B3. My child likes to read about science even 
when the teacher doesn’t ask him/her to. 


□2 

□3 

□4 

B4. Science is easy for my child. 

□1 

□2 

□3 

□4 

B5. When my child hears the word “science,” 
he/she has a feeling of dislike. 

□, 

□2 

□3 

□4 

B6. Most people should study some science. 

□1 

□2 

□3 

□4 

B7. Sometimes my child reads ahead in the 
science book he/she uses in school. 

□1 

□2 

□3 

□4 

B8. 1 don’t think my child could do advanced 

science. 

□1 

□2 

□3 

□4 

B9. 1 think my child could do more difficult 

science work than what he/she does now. 

□1 

□2 

CO 

□ 

□4 

B10. Science is helpful in understanding today’s 
world. 

□1 

□2 

□3 

□4 

B1 1 . My child usually understands science when it 
is talked about in class at school. 

□1 

□2 

□3 

□4 

B12. No matter how hard he/she tries, my child 
cannot understand science. 

□1 

□2 

CO 

□ 

□4 

B1 3. My child feels tense when someone talks to 
him/her about science. 

□1 

□2 

□3 

□4 

B14. My child likes to understand the scientific 
explanation for things. 

□1 

□2 

□3 

□4 


Please go to the next page. 


541 / 

542/ 

543/ 

544/ 

545/ 

546/ 

547/ 

548/ 

549/ 

550/ 

551/ 

552/ 

553/ 

554/ 
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Check one box to tell us if you Strongly Disagree, Disagree, Agree, or Strongly Agree with each 
statement. 


Statements 

Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

B15. My child often thinks, “1 cannot do this,” when 
a science assignment seems hard. 

□i 

□ 2 

□3 

□4 

B16. When my child can’t immediately solve a 
science problem, he/she sticks with it until a 
solution is reached. 

□i 

□2 

□3 

□4 

B1 7. Science is of great importance to a country's 
development. 

□i 

□2 

□3 

□4 

B1 8. It is important to know science in order to get 
a good job. 

□i 

□2 

□3 

□4 

B19. My child likes the challenge of science 
assignments. 

□i 

□2 

□3 

□4 

B20. It makes my child nervous to even think about 
doing science. 

□i 

□2 

□3 

□4 

B21 . It scares my child to have to take a science 
class. 

□i 

□2 

□3 

□4 

B22. My child thinks it is important to understand 
the schoolwork he/she does in science. 

□i 

□2 

□3 

□4 

B23. My child has a good feeling about science. 

□i 

□2 

□3 

□4 

B24. Science is one of my child’s favorite subjects. 

□i 

□2 

□3 

□4 

B25. My child really wants to learn about science. 

□i 

□2 

□3 

□4 

B26. My child does not do very well in science. 

□i 

□2 

□3 

□4 

B27. Sometimes my child wants to learn more 
about science than the teacher covers at 
school. 

□i 

□2 

□3 

□4 

B28. My child thinks science is boring. 

□i 

□2 

□3 

□4 


C. Out of School Activities 


Below is a list of activities that you and your child may or may not have shared over the past three 
months. 


Over the past three months, how often did you and your child do this activity together? (Check only 
one box for each activity.) 



Not at all 
over the 
past 
three 
months 

One 

time 

About 
once a 
month 

About 
twice a 
month 

About 
once a 
week 

More 
than 
once a 
week 

Cl. 

Go to a science museum or zoo 

□, 

□2 

□3 

□4 

□5 

□a 

C2. 

Read about science in a book, 
magazine, or website 


□2 

□3 

□4 

□5 

□a 

C3. 

Have your child take something 
apart, put something together or 
build something new 


□2 

□3 

□4 

□5 

□a 

C4. 

Watch a science program on TV 


□2 

□3 

□4 

□5 

□a 

C5. 

Try to figure out how something 
works 


□2 

□3 

□4 

□5 

□e 

C6. 

Do an experiment 


□2 

□3 

□4 

□5 

□e 

C7. 

Have your child learn how to use a 
new tool, a new piece of 
equipment, or a new scientific 
instrument 


□2 

□3 

□4 

□5 

□ a 

C8. 

Use a science kit or do a science 
project at home 

□1 

□2 

□3 

□4 

□5 

□a 

C9. 

Design something using your 
child’s ideas 


□2 

□3 

□4 

□5 

□ a 

CIO. 

Play a video game 

□1 

□2 

□3 

□4 

□5 

□ a 

Cl 1 . 

Watch a movie at home or in a 
theater 


□2 

□3 

□4 

□5 

□ a 

C12. 

Use a computer or technology to 
solve a problem 

□1 

□2 

□3 

□4 

□5 

□ a 


Please go to the next page. 


Please go to the next page. 


569/ 

570/ 

571/ 

572 / 
573/ 
574/ 

575/ 

576/ 

577/ 

578/ 

579/ 

580 / 
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C13. 


During the past three months, did your child participate in any of the following 
science or mathematics activities? 


Yes 

No 



□ i 

□ 2 

A science fair at school 

1060/ 

□ i 

□ 2 

A science or mathematics club at school 

1061/ 

□ i 

□ 2 

Study with a science or mathematics tutor 

581/ 

□ i 

□ 2 

Tutor someone else in science or mathematics 

582/ 583/B 

□ i 

□ 2 

Participate in a science or math program other than NASA’s SEMAA: 

584/ 



(What is the name of this program? 

) 585-634/ 



(Who is the sponsor of this program? 

) 635-684/ 

□ i 

□ 2 

Participate in any other science or mathematics activities 

685/ 



Please describe: 

686-735/B 


890-1039/ 


In the next 3 months, how often do you think that you and your child together will do each of these 
activities? Check only one box for each activity. 



Not at all 
in the 
next 3 
months 

One 

time 

About 
once a 
month 

About 
twice a 
month 

About 
once a 
week 

More 
than 
once a 
week 

736/ 

737/ 

738/ 

739/ 

740/ 

741/ 

742/ 

743/ 

744/ 

745/ 

746/ 

747/ 

Cl 4. Go to a science museum or zoo 

□, 

□2 

□3 

□4 

□5 

□e 

Cl 5. Read about science in a book, 
magazine, or on a website 

□1 

□2 

□3 

□4 

□5 

□a 

Cl 6. Have your child take something 
apart, put something together or 
build something new 


□2 

□3 

□4 

□5 

□a 

Cl 7. Watch a science program on TV 

□1 

□2 

□3 

□4 

□5 

□a 

Cl 8. Try to figure out how something 
works 


□2 

□3 

□4 

□5 

□a 

Cl 9. Do an experiment 

□1 

□2 

□3 

□4 

□5 

□e 

C20. Have your child learn how to use a 
new tool, a new piece of 
equipment, or a new scientific 
instrument 


□2 

□3 

□4 

□5 

□a 

C21 . Use a science kit or do a science 
project at home 

□1 

□2 

□3 

□4 

□5 

□a 

C22. Design something using your 
child's ideas 


□2 

□3 

□4 

□5 

□a 

C23. Play a video game 


□2 

□3 

□4 

□5 

□a 

C24. Watch a movie at home or in a 
theater 


□2 

□3 

□4 

□5 

□e 

C25. Use a computer or technology to 
solve a problem 

□1 

□2 

□3 

□4 

□5 

□a 


Please go to the next page. 
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I). Science and Math in School 

D 1 . My child’s current grade in science class is (select the one box that best matches 

the grade your child is currently receiving): 748/ 

d "A” or Excellent 
d 2 “B” or Good 

□ ■ C” or Satisfactory 

□ ■ D” or Barely Passing 

□ ■ F” or Failing 
^6 I don’t know 

□ Not applicable, my child is not receiving a science grade this year 

D 2 . I believe my child’s ability in science is... (Check only one box) 749/ 

□ Much higher than other children his/her age 

□ Somewhat higher than other children his/her age 

□ About the same as other children his/her age 
□4 Somewhat lower than other children his/her age 

□ Much lower than other children his/her age 

D 3 . My child’s current grade in math class is (select the one box that best matches 

the grade your child is currently receiving): 750/ 

□ 1 “A” or Excellent 
Q 2 “B” or Good 

□ ■ C” or Satisfactory 
□ 4 “ D” or Barely Passing 

□ ■ F” or Failing 
^6 I don’t know 

□ Not applicable, my child is not receiving a math grade this year 

D 4 . I believe my child’s ability in math is... (Check only one box) 751/ 

□ Much higher than other children his/her age 

□ Somewhat higher than other children his/her age 

□ About the same as other children his/her age 

□ Somewhat lower than other children his/her age 

□ 5 Much lower than other children his/her age 


Please go to the next page. 
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When your child reaches high school, there may be classes that you would encourage your child to 
take, even if the class is not required. 


How much would you encourage your child to take each of the following classes in high school? 
Please check only one box for each class. 


HIGH SCHOOL CLASSES 

Would 

definitely not 
encourage 

Would 
probably not 
encourage 

Would 

probably 

encourage 

Would 

definitely 

encourage 

El. Physics 

□i 

□» 

□3 

□4 

E2. Biology 

□i 

□2 

□3 

□4 

E3. Chemistry 


□2 

□3 

□4 

E4. Art 


□2 

□3 

□4 

E5. Earth Science 


□2 

□3 

□4 

E6. Astronomy 


□2 

□3 

□4 

E7. Computer science 

□i 

□2 

□3 

□4 

E8. Anatomy & Physiology 

□i 

□2 

□3 

□4 

E9. Music 


□2 

□3 

□4 

E10. Advanced Physics 


□2 

□3 

□4 

Ell. Advanced Biology 

□i 

□2 

□3 

□4 

El 2. Advanced Chemistry 


□2 

□3 

□4 

El 3. Algebra 

□i 

□2 

□3 

□4 

El 4. Geometry 


□2 

□3 

□4 

El 5. Trigonometry 


□2 

□3 

□4 

El 6. Calculus 


□2 

□3 

□4 

El 7. Statistics 

□i 

□2 

□3 

□4 

El 8. History 


□2 

□3 

□4 


752/ 

753/ 

754/ 

755/ 

756/ 

757/ 

758/ 

759/ 

760/ 

761/ 

762/ 

763/ 

764/ 

765/ 

766/ 

767/ 

768/ 

769/ 


Please go to the next page. 
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Please check one box to tell us if you Strongly Disagree, Disagree, Agree, or Strongly Agree with 
each statement. 



Strongly 

Disagree 

Disagree 

Agree 

Strongly 

Agree 

El 9. When my child is an adult, he/she will 
need a good understanding of basic 
science. 

□i 

□ 2 

□3 

□4 

E 20 . When my child is an adult, he/she will 
need a good understanding of 

advanced science. 

□i 

□2 

□3 

□4 

E 21 . When my child is an adult, he/she will 
need a good understanding of basic 
mathematics. 

□i 

□2 

□3 

□4 

E 22 . When my child is an adult, he/she will 
need a good understanding of 

□i 

□2 

□3 

□4 


advanced mathematics. 


Please go to the next page. 
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If your child goes to college, he or she will have to choose a major field of study in which to earn a 
college degree. 

If your child were interested in many different fields, how much would you encourage your child to 
earn a degree in each of the following? Please check only one box for each type of degree. 

774/ 
775/ 
776/ 
777/ 
778/ 
779/ 
780/ 
781/ 
782/ 
783/ 
784/ 
785/ 
786/ 
787/ 

Please go to the next page. 


COLLEGE DEGREE 

Would 

definitely not 
encourage 

Would 
probably not 
encourage 

Would 

probably 

encourage 

Would 

definitely 

encourage 

E 23 . Physics 

□1 

□2 

□3 

□4 

E 24 . Biology 

□1 

□2 

□3 

□4 

E 25 . Computer science 


□2 

□3 

□4 

E 26 . Chemistry 


□2 

□3 

□4 

E 27 . Literature (books and writing) 


□2 

□3 

□4 

E 28 . History 


□2 

□3 

□4 

E 29 . Engineering 

□1 

□2 

□3 

□4 

E 30 . Mathematics 


□2 

□3 

□4 

E 31 . Art, music, or acting 


□2 

□3 

□4 

E 32 . Medicine 


□2 

□3 

□4 

E 33 . Business, accounting, or 
finance 

□1 

□2 

□3 

□4 

E 34 . Education 


□2 

□3 

□4 

E 35 . Law 


□2 

□3 

□4 

E 36 . Astronomy 


□2 

□3 

□4 
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